Such geek. Wow.

Replacing the Tuya components from a Mercator Ikuü light switch

Wed, 04 Dec 2024 00:00:00 +1300

TL;DR

You can replace the Tuya WB3S with an Espressif ESP-12F (or ESP-12E) and ESPHome.
You need to pull EN up to 3.3v, and pull GPIO15 down to ground via a 10k resistor.
If you’re on Linux, make sure your user is in the dialout group to write to the ESP-12.
Scroll to the end for the YAML.
- Comment out the appropriate buttons for your model.
- I’ve confirmed that SSW01G-WIFI uses button 2, and SSW02G-WIFI uses buttons 1 and 3. Presumably SSW03G-WIFI uses all of them, but I don’t have one to confirm.
- Make sure you change light.toggle into fan.toggle if you decide to declare a button as a fan, etc.

The article

The Mercator Ikuü range of smart devices is another rebranded Tuya line. As a result, controlling the device with Home Assistant was slow due to the calls to the Tuya datacentre — and honestly I just wanted to have devices that are entirely under my control. I started by disassembling the switch and immediately saw the Tuya WB3S on the underside of the top plate.

The switch, opened up. The controlling PCB sits on the underside of the top plate.

Be careful lifting this PCB out, as there are pins connecting it to the 240v plane — make sure you lift it straight. This set of connecting pins is labelled CN1, and is responsible for connecting the controlling MCU to the 240v relays.

On the board is the Tuya WB3S, and a Holtek BS813A-1 touch-button sensor (https://www.holtek.com.tw/page/vg/BS81xA-x). It does not appear to contain a TuyaMCU chip. The same board is used for the SSW01G-WIFI, SSW02G-WIFI, and SSW03G-WIFI models — it seems the SSW04G-WIFI is a different board.

The Holtek BS813A-1 touch sensor pinout.

The way this Holtek sensor works is, while a touch is sensed by the Key1 leg, it will pull Kout1 down to zero volts. This is momentary and returns to 3.3v once the touch is removed. The MCU listens for the appropriate pin to be pulled low, and triggers a relay in response.

The controller board. The red arrow marks where I (incorrectly) oriented my CN1 numbering from.

The red arrow is where I (incorrectly) oriented my CN1 naming from. I’ll refer to the CN1 pins in reading order; pins 1-5 on the top row, pins 6-10 on the bottom. After writing this, I realised that this is almost certainly backwards: the bottom-right pin — what I call pin 10 — has a square base, indicating it should have been pin 1. Oh well.

Here’s the pinout of the Tuya WB3S, from https://docs.libretiny.eu/boards/wb3s/:

The Tuya WB3S pinout, via LibreTiny.

Note, the grey numbers in the WB3S pinout are simply positions; the pin names are burgundy (i.e., P23 is in position 2). I can see PCB traces at positions 15, 14, 13, 12, 10, 9 (GND), and 8 (3.3v). Positions 17-22 are visibly unsoldered.

WB3S GND (position 9) goes to CN1 Pins 1 & 2
WB3S 3.3v (position 8) goes to CN1 Pin 6
WB3S P6 (position 7) goes to CN1 Pin 7, via resistor R7, and on to LED1
WB3S P7 (position 10) goes to CN1 pin 9, via resistor R14, and on to LED3
WB3S P8 (position 14) goes to capacitor C2 and onto CN1 pins 1 & 2
WB3S P9 (position 13) goes to CN1 pin 8, via resistor R13 and LED2 (empty on this model)
WB3S P10 (position 15) goes to resistor R2, which is empty on this model
WB3S P24 (position 6) goes to resistor R11, which goes on to LED4 (plate lighting)

And the inverse to cross-check myself: on CN1, I can see upper PCB traces at 1, 2, 6, 7 and 8 — plus an underside trace to 9.

CN1 Pin 1 goes to WB3S GND (position 9)
CN1 Pin 2 goes to WB3S GND (position 9)
CN1 Pin 6 goes to WB3S 3.3v (position 8)
CN1 Pin 7 goes to WB3S P6 (position 7) via resistor R7, which then goes to LED1
CN1 Pin 8 goes to WB3S P9 (position 13) — this also traces to R13, which is empty on this model, and appears to be for the absent middle switch LED2
CN1 Pin 9 goes to WB3S P7 (position 10) via resistor R14, and onward to LED3

I double-checked this by powering up the WB3S chip and checking the voltage on the pins as I hit the buttons. All seems correct so far. Before I desoldered the WB3S on the board, I connected RX and TX and dumped the Tuya firmware, and decoded this with the OpenBeken flash tool from https://github.com/openshwprojects/BK7231GUIFlashTool. Here are the interpreted results:

Device configuration, as extracted from Tuya:
- Button (channel 1) on P24 [pinout position 6]
- Relay (channel 2) on P7 [pinout position 10]
- Button (channel 2) on P8 [pinout position 14]
- Relay (channel 1) on P6 [pinout position 7]

Device seems to be using WB3S module, which is using BK7231T.
And the Tuya section starts, as usual, at 2023424

These results didn’t align with my original tracing, so it was really lucky I did this. Although I had correctly identified the relays in position 7 & 10, I had misunderstood how the buttons work. It was thanks to this discrepancy that I started testing the levels while touching the touch sensors, and discovered that button-presses cause the voltage to drop on P8 (position 14) and P24 (position 6). This led me to read up on how the Holtek chip works, helping me to understand what’s going on here!

So, indeed we have two buttons and two relays. The buttons are held HIGH while untouched, and momentarily drop to LOW when touch is detected. In turn, these state changes trigger toggles to the relay outputs. I can finally start writing YAML!

I’m replacing this Tuya WB3S with an Espressif ESP12-F, which is largely a perfect drop-in replacement:

The ESP-12 pinout.

Translating from the WB3S, it appears:

P24 (position 6 on the WB3S) maps to GPIO12 on the ESP12
P7 (position 10 on the WB3S) maps to GPIO15 on the ESP12
P8 (position 14 on the WB3S) maps to GPIO5 on the ESP12
P6 (position 7 on the WB3S) maps to GPIO13 on the ESP12
P10 (position 15 on the WB3S) maps to RX on the ESP12
Additionally, EN needs to be pulled up to 3.3v, and GPIO15 needs to be pulled down to ground via a 10k resistor.

This is my first time programming an ESP12 (my previous ESPHome devices have been ESP32’s), and I did find them a little more difficult — I ended up bumping into a permissions problem, and thanks to this post I was able to resolve the error failed to execute 'open' on 'SerialPort': Failed to open serial port. The solution was either to add myself to the dialout group, or set the ACL on ttyUSB0 (only a transient workaround):

sudo setfacl -m u:USERNAME:rw /dev/ttyUSB0
sudo usermod -aG dialout USERNAME

Desoldering the SMT WB3S was enough of a challenge that I asked a friend for help, and he did the first couple for me while I waited for a hot plate to arrive from AliExpress. Then I did the third one and only slightly cooked my board.

The ESP-12F in place, complete with the 10k pull-down bodge resistor.

Result! Not only is this 100% locally-controlled and forever disconnected from the Tuya Cloud, but the response time is ridiculously fast now. What used to be 200~2000ms response time (yes really, depending on network congestion), is now closer to 20ms (unmeasurably fast with my zero instruments).

The YAML snippet

The YAML for the one-gang, two-gang, and three-gang versions is below. Note this is partial YAML; other important parts such as enabling the API and OTA are left out, but I’ll include them below for new folks.

esp8266:
 board: esp01_1m

binary_sensor:
 # Button 1 - not present on the SSW01G
 - platform: gpio
 id: button_1
 pin:
 number: GPIO12
 inverted: true
 mode: INPUT_PULLUP
 on_press:
 then:
 - light.toggle: switch_1
 internal: True

 # Button 2 - not present on the SSW02G
 - platform: gpio
 id: button_1
 pin:
 number: RX
 inverted: true
 mode: INPUT_PULLUP
 on_press:
 then:
 - light.toggle: switch_2
 internal: True

 # Button 3 - not present on the SSW01G
 - platform: gpio
 id: button_3
 pin:
 number: GPIO5
 inverted: true
 mode: INPUT_PULLUP
 on_press:
 then:
 - light.toggle: switch_3
 internal: True

output:
 # Relay 1 - not present on the SSW01G
 - platform: gpio
 id: relay_1
 pin: GPIO13

 # Relay 2 - not present on the SSW02G
 - platform: gpio
 id: relay_2
 pin: GPIO4

 # Relay 3 - not present on the SSW01G
 - platform: gpio
 id: relay_3
 pin: GPIO15

# Expose the relays as lights and/or fans
light:
 # Light 1 - not present on the SSW01G
 - platform: binary
 name: Switch 1
 output: relay_1
 id: switch_1

 # Light 2 - not present on the SSW02G
 - platform: binary
 name: Switch 2
 output: relay_2
 id: switch_2

 # Light 3 - not present on the SSW01G
 - platform: binary
 name: Switch 3
 output: relay_3
 id: switch_3

# Example of having a fan on button 3 instead - if you do this, you must change
# the earlier matching button declaration from light.toggle to fan.toggle.
fan:
 - platform: binary
 # Button 3
 name: Ensuite Fan
 output: relay_3
 id: switch_3

That’s it! Upload that fragment into your normal ESPHome setup, and you should be in business.

In case you’re new to this, below is my standard YAML — this would go at the very top of your YAML code block within your ESPHome device declaration, above the previous stuff. This enables a variety of important bits, plus adds helpful sensors for WiFi info and uptime.

esphome:
 name: mercator-ikuu
 friendly_name: "Mercator Ikuu Smart Switch"

# Enable logging
logger:

# Enable Home Assistant API
api:
 encryption:
 key: # for you to fill in

# Enable over-the-air updates; requires you to have a saved secret
ota:
 - platform: esphome
 password: !secret ota_password

# Your wifi connection details; requires you to have saved secrets
wifi:
 ssid: !secret wifi_ssid
 password: !secret wifi_password
 # Enable open fallback hotspot if wifi connection fails
 ap:
 {}

# Enables a helpful web portal if wifi connection fails
captive_portal:

# Allows you to browse to the device and have a direct view of what's happening
web_server:

# Displays wifi signal strength and device uptime
sensor:
 - platform: wifi_signal
 name: Wifi Signal Strength
 update_interval: 60s
 - platform: uptime
 name: Uptime

# Displays network IP address, connected SSID and BSSID, and MAC
text_sensor:
 - platform: wifi_info
 ip_address:
 name: IP
 ssid:
 name: SSID
 bssid:
 name: BSSID
 mac_address:
 name: MAC

task pve-bridge blocked for more than 120 seconds - on Proxmox with MDATP

Sun, 13 Mar 2022 00:00:00 +1300

Today I updated and rebooted a bunch of servers - all in a day’s work. Everything went fine, until one of the hosts (the big one) didn’t boot up again properly.

I’ve got two hosts, both running ProxmoxVE (version 7.1) and Microsoft Defender for Endpoint (version 101.61.69-insiderfast). On reboot, one of the hosts came up fine; the other didn’t.

Long story short, uninstalling mdatp fixed the problem - wild hypothesis, but I suspect it was interfering with either network bridge or interprocess communication.

The rest of this post is technical info for people who might be able to fix it, and so that this blog post will come up for the other people who inevitably bump into this!

Symptoms - upon boot, the host came up, was apparently able to launch one VM, and recieved a timeout for all subsequent VM’s:

The lines are truncated, but all those pink lines are showing TASK ERROR: start failed: command '/usr/bin/kvm -id 153 [...]' failed: got timeout.

I say ‘apparently’ launched one VM, because I was unable to interact with it - in fact all the launch-on-boot VM’s appeared to have booted, except everything about them was non-responsive, including Proxmox stats windows and console.

Trying to manually interact with the VM’s using qm failed entirely - the console froze, completely unresponsive to either ^C or ^X, until I forcefully disconnected from SSH.

Checking dmesg revealed a horror story:

Mar 13 16:54:36 buckwheat kernel: [ 159.853700] vmbr1: port 1(eth3) entered blocking state
Mar 13 16:54:36 buckwheat kernel: [ 159.853703] vmbr1: port 1(eth3) entered forwarding state
Mar 13 16:54:43 buckwheat kernel: [ 166.700586] ixgbe 0000:81:00.1 eth3: NIC Link is Down
Mar 13 16:54:43 buckwheat kernel: [ 166.701135] vmbr1: port 1(eth3) entered disabled state
Mar 13 16:54:45 buckwheat kernel: [ 168.294208] ixgbe 0000:81:00.1 eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX
Mar 13 16:54:45 buckwheat kernel: [ 168.294343] vmbr1: port 1(eth3) entered blocking state
Mar 13 16:54:45 buckwheat kernel: [ 168.294349] vmbr1: port 1(eth3) entered forwarding state
Mar 13 16:54:47 buckwheat pvestatd[5040]: VM 112 qmp command failed - VM 112 qmp command 'query-proxmox-support' failed - unable to connect to VM 112 qmp socket - timeout a>
Mar 13 16:54:50 buckwheat pvestatd[5040]: VM 105 qmp command failed - VM 105 qmp command 'query-proxmox-support' failed - unable to connect to VM 105 qmp socket - timeout a>
Mar 13 16:54:53 buckwheat pvestatd[5040]: VM 115 qmp command failed - VM 115 qmp command 'query-proxmox-support' failed - unable to connect to VM 115 qmp socket - timeout a>
Mar 13 16:54:55 buckwheat pve-guests[11257]: start failed: command '/usr/bin/kvm -id 153 -name Eden -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/153.qmp,s>
Mar 13 16:54:56 buckwheat pvesh[5530]: Starting VM 153 failed: start failed: command '/usr/bin/kvm -id 153 -name Eden -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qem>
Mar 13 16:54:56 buckwheat pve-guests[5530]: <root@pam> end task UPID:buckwheat:000015C7:00000710:622D6A6F:startall::root@pam: OK
Mar 13 16:54:56 buckwheat systemd[1]: Finished PVE guests.
Mar 13 16:54:56 buckwheat systemd[1]: Reached target Multi-User System.
Mar 13 16:54:56 buckwheat systemd[1]: Reached target Graphical Interface.
Mar 13 16:54:56 buckwheat systemd[1]: Starting Update UTMP about System Runlevel Changes...
Mar 13 16:54:56 buckwheat systemd[1]: systemd-update-utmp-runlevel.service: Succeeded.
Mar 13 16:54:56 buckwheat systemd[1]: Finished Update UTMP about System Runlevel Changes.
Mar 13 16:54:56 buckwheat systemd[1]: Startup finished in 4.603s (kernel) + 2min 54.823s (userspace) = 2min 59.427s.
Mar 13 16:54:56 buckwheat pvestatd[5040]: VM 104 qmp command failed - VM 104 qmp command 'query-proxmox-support' failed - unable to connect to VM 104 qmp socket - timeout a>
Mar 13 16:54:59 buckwheat pvestatd[5040]: VM 153 qmp command failed - VM 153 qmp command 'query-proxmox-support' failed - got timeout
Mar 13 16:54:59 buckwheat pvestatd[5040]: status update time (30.177 seconds)
Mar 13 16:55:18 buckwheat pvestatd[5040]: VM 104 qmp command failed - VM 104 qmp command 'query-proxmox-support' failed - unable to connect to VM 104 qmp socket - timeout a>
Mar 13 16:55:21 buckwheat pvestatd[5040]: VM 153 qmp command failed - VM 153 qmp command 'query-proxmox-support' failed - unable to connect to VM 153 qmp socket - timeout a>
Mar 13 16:55:24 buckwheat pvestatd[5040]: VM 112 qmp command failed - VM 112 qmp command 'query-proxmox-support' failed - unable to connect to VM 112 qmp socket - timeout a>
Mar 13 16:55:27 buckwheat pvestatd[5040]: VM 105 qmp command failed - VM 105 qmp command 'query-proxmox-support' failed - unable to connect to VM 105 qmp socket - timeout a>
Mar 13 16:55:30 buckwheat pvestatd[5040]: VM 115 qmp command failed - VM 115 qmp command 'query-proxmox-support' failed - unable to connect to VM 115 qmp socket - timeout a>
Mar 13 16:55:30 buckwheat pvestatd[5040]: status update time (30.186 seconds)
Mar 13 16:55:48 buckwheat pvestatd[5040]: VM 153 qmp command failed - VM 153 qmp command 'query-proxmox-support' failed - unable to connect to VM 153 qmp socket - timeout a>
Mar 13 16:55:51 buckwheat pvestatd[5040]: VM 104 qmp command failed - VM 104 qmp command 'query-proxmox-support' failed - unable to connect to VM 104 qmp socket - timeout a>
Mar 13 16:55:54 buckwheat pvestatd[5040]: VM 115 qmp command failed - VM 115 qmp command 'query-proxmox-support' failed - unable to connect to VM 115 qmp socket - timeout a>
Mar 13 16:55:57 buckwheat pvestatd[5040]: VM 105 qmp command failed - VM 105 qmp command 'query-proxmox-support' failed - unable to connect to VM 105 qmp socket - timeout a>
Mar 13 16:56:00 buckwheat pvestatd[5040]: VM 112 qmp command failed - VM 112 qmp command 'query-proxmox-support' failed - unable to connect to VM 112 qmp socket - timeout a>
Mar 13 16:56:00 buckwheat pvestatd[5040]: status update time (30.191 seconds)
Mar 13 16:56:00 buckwheat kernel: [ 243.301563] INFO: task wdavdaemon:5583 blocked for more than 120 seconds.

The wdavdaemon is shown blocking here, but pve-bridge turned up just as frequently in the logs:

root@buckwheat:/var/log# grep blocked syslog
Mar 13 18:34:14 buckwheat kernel: [ 242.802463] INFO: task wdavdaemon:5740 blocked for more than 120 seconds.
Mar 13 18:34:14 buckwheat kernel: [ 242.802658] INFO: task wdavdaemon:4899 blocked for more than 120 seconds.
Mar 13 18:34:14 buckwheat kernel: [ 242.802840] INFO: task pve-bridge:6117 blocked for more than 120 seconds.
Mar 13 18:34:14 buckwheat kernel: [ 242.802945] INFO: task pve-bridge:7734 blocked for more than 120 seconds.
Mar 13 18:34:14 buckwheat kernel: [ 242.803057] INFO: task pve-bridge:9120 blocked for more than 120 seconds.
Mar 13 18:34:14 buckwheat kernel: [ 242.803170] INFO: task pve-bridge:10283 blocked for more than 120 seconds.
Mar 13 18:36:15 buckwheat kernel: [ 363.631430] INFO: task wdavdaemon:5740 blocked for more than 241 seconds.
Mar 13 18:36:15 buckwheat kernel: [ 363.631610] INFO: task wdavdaemon:4899 blocked for more than 241 seconds.
Mar 13 18:36:15 buckwheat kernel: [ 363.631776] INFO: task pve-bridge:6117 blocked for more than 241 seconds.
Mar 13 18:36:15 buckwheat kernel: [ 363.631872] INFO: task pve-bridge:7734 blocked for more than 241 seconds.

So the interesting thing is that another node - foxtail - was also updated to the same version of both Proxmox and mdatp. The only obvious difference I can see are the numbers of VLANs each of them are bridging to:

This is a list of the bridges now, when they’re both working (sorry, Markdown hates tabs; the below should be four tidy columns):

root@foxtail:/var/log# brctl show
bridge name	bridge id		STP enabled	interfaces
fwbr101i0	8000.9e7164f22bab	no		fwln101i0
							tap101i0
vmbr0		8000.c2602668d5f4	no		eno1
							fwpr101p0
							tap100i0
							tap109i0

versus

root@buckwheat:/var/log# brctl show
bridge name	bridge id		STP enabled	interfaces
fwbr102i0	8000.6	abd66965438	no		fwln102i0
							tap102i0
fwbr104i0	8000.f60b3f1a3118	no		fwln104i0
							tap104i0
fwbr105i0	8000.961281d6d5dc	no		fwln105i0
							tap105i0
vmbr0		8000.90e2bad572a8	no		eth2
							tap112i0
							tap115i0
							tap153i0
vmbr0v5		8000.0ea35ee013ba	no		eth2.5
							fwpr104p0
vmbr0v666	8000.1a23193ccf17	no		eth2.666
							fwpr102p0
							fwpr105p0
vmbr1		8000.90e2bad572a9	no		eth3

But when buckwheat was broken, the majority of the bridges were not visible:

At some point during troubleshooting, I restarted pvedaemon - although the service had started correctly, the restart failed with error: timeout waiting on systemd.

Goldenhorse - Wake Up Brother bass tab

Sat, 19 Feb 2022 00:00:00 +1300

I know Goldenhorse didn’t get particularly huge, but I always really enjoyed their work. Wake Up Brother is one of those ones that really feels like an achievable song for a small band - it’s got solid, driving bass; a groovy bright guitar line; and requires quite a dynamic singer to pull it off.

Some friends and I are planning to add it to our band’s repertoire, but the bassline is a bit too complex for us to transcribe properly. And the song wasn’t particularly well-known, so there’s really only one public bass tab, and it wasn’t really comprehensive enough… So I turned to Fiverr!

There are a bunch of folks there who will transcribe bass tab - but one of them stuck out to me, so I decided to work with georgex0006.

I was super impressed with his speed and quality of work. I never could have nailed the bassline as well as he did, and providing a playalong video as well is 100% awesome.

I’m sharing the bass tab and playalong here, with his permission. If you need someone to do some bass tab for you, please give him your support!

Tab & score PDF Version here
Playalong MP4 Version here

And for my personal use (and yours, if you own the album), I purchased a .FLAC of the track which is stored here. However if you don’t already own it, please pay for it yourself first, or listen to it on YouTube where they can deal with copyright issues.

Seltzer brewing

Sun, 12 Dec 2021 00:00:00 +1300

I got into brewing seltzer during the spring of 2021, and these are my notes. They’re in descending order, so the most-recent are at the top. The notes got progressively less-detailed as time went by, so those looking to actually learn things (e.g. “How much yeast nutrient should I use in a seltzer that’s 100% dextrose”) should read the highlights and then read from bottom-up. :)

Highlights:

Kveik Voss is fast-fermenting and relatively transparent flavour. Excellent for seltzer
It does take a couple days post-fining for the yeast scent to diminish
Recycling yeast is tricky with seltzer due to all the fruit sediment
With only sugars and fruit flesh, yeast nutrient is essential - about 15 grams seems to do the trick
One kg of dextrose in 19 litres will raise the ABV by approximately 2.65%
Don’t boil the fruit or you’ll get pectin haze; raise it to about 75°C to sterilise it, though
Table sugar is a perfectly passable sugar; make invert sugar by boiling it with a little citric acid, 20-60 minutes
Citric acid is amazing

Fruit notes:

Mashing the fruit a little bit is a significant factor in releasing the fruit flavours
Lychee have quite a prominent character, good to pair with something that can compete
Mandarins will impart very little flavour unless you mash them really good pre-ferment
Jackfruit is almost undetectable, even with two cans; will try blending next time
Pineapple seems to digest(?) other fruits. The other fruit becomes quickly less-noticable while the pineapple becomes more prominent
Raspberries create a visually beautiful brew, as do plums
Passionfruit pulp is pretty disappointing; in brew #7 I used THREE cans, and it was nearly undetectable
Mango is pretty awful, really - lots of fruit mush, lots of haze, some weird aged-fruit flavour… avoid imho

Yeast Awakener

To reawaken the yeast from the fridge, I make a mix of:

A tablespoon of sugar (any variety, often dextrose)
10 grams chopped raisins OR 2tsp lemon juice
6 grams Marmite
Pinch of yeast nutrient

Once those are dissolved together, I pour in collected yeast from the previous batch.

Other thoughts:

2022-01-04 - my brews are still coming out very sweet - I’m yearning for something with a dryer character. Hard seltzer was featured in an exBEERiment on Brulosophy on the difference between dextrose and sucrose (it seems table sugar will be fine!), where Phil mentions using the W04 Paramount wine yeast(!), along with 32g of yeast nutrient(!!). Perhaps this will solve my sweet brews? Might also get my water acidity tested to make sure it’s at a good pH for the fermentation.
2022-01-21 - the increased yest nutrient is working out very well; much cleaner and dryer - 25g seems good. Ordered some Fermaid AT today as well to see if a slower release of yeast-available nitrogen (YAN) will support the yeast better

Recipes and notes

Brew #11, 2022-03-12 - brewing: Blueberry and Mandarin Seltzer - large edition

Recipe:

3kg invert table sugar
10g Citric acid
15g+15g Fermaid-AT
Kveik Voss
310g tinned mandarins
1275g tinned blueberries

Notes:

Decided it was high time to brew a larger batch - this is a 25 litre brew
The citric acid has been amazing - boosted the quantity in this one to compensate for the larger proportion of fruit
Dosed 15g of Fermaid-AT at inital brew; will dose another 15g at 3 days
Looking to try some amylase enzyme to see if I can really dry out the seltzer

Brew #10, 2022-02-20 - 2022-02-28: Blueberry and Mandarin Seltzer

Recipe:

2kg white table sugar (see notes)
5g Citric acid
10g+10g yeast nutrient (Fermaid-AT)
A fresh Kveik Voss
310g tinned mandarins
850g tinned blueberries

Notes:

My Fermaid-AT has arrived! It’s used at a rate up to 1g/l in low-YAN environments, so used 10g and will add another 10g in 2 days
Two batches of 10g FermaidAT was pretty good, the Kveik chews that stuff up super-quick, literally over 36 hours
I really liked this combination so revisiting the double-blueberry and mandarin with a couple tweaks
The invert sugar experiment seemed to be a success, so 2kg table sugar + 5g citric acid; boil for 20mins
Easily one of my favourites - possibly THE favourite - so far. 4/5, only minor improvements
Update, citric acid is amazing, absolutely brings the zing that I was craving. Still getting sweet brews but much more balanced now
Next brew will try 15g+10g nutrient

Brew #9, 2022-01-26 - 2022-02-01: Apple, Mango, and Guava Seltzer

Recipe:

2kg white table sugar (see notes)
4g Citric acid
23g yeast nutrient
My weird hybrid yeast now (see notes)
1kg diced apple
410g tinned guava
850g mango pulp

Notes:

I ran out of yeast nutrient so only 23g this time
Overshot water, but had lots more fruit than normal so this is a 24l batch
Decided to attempt invert sugar instead of dextrose! 2kg table sugar + 4g citric acid; boil for 20mins
The Kveik Voss + Mangrove Jack’s Seltzer yeast is doing the job very nicely
Initial flavour suggests this might be a goodie.
Final flavour - meh. 2/5, I wouldn’t brew this again. The citric acid seemed to be a good call though.

Brew #8, 2022-01-16 - 2022-01-21: Blueberry and Mandarin Seltzer

Recipe:

1.7kg Dextrose, 300g white sugar
Kveik Voss plus Mangrove Jacks’ Seltzer yeast
25g yeast nutrient
310g tinned mandarins
850g tinned blueberries

Notes:

Previous run went very well (much drier result), so bumped nutrient to 25g!
Recycled the Kveik Voss + MJ’s Seltzer yeast from last batch; seemed pretty okay
Result - an absolute banger! Really delicious; next time I’ll replace the mandarin with something more acidic though

Brew #7, 2022-01-05 - 2022-01-10: Pineapple Passion Seltzer

Recipe:

2kg Dextrose
Kveik Voss plus Mangrove Jacks’ Seltzer yeast
20g yeast nutrient
420g tinned crushed pineapple
510g tinned passionfruit pulp

Notes:

Increased yeast nutrient to 20g this time, will see if it makes a difference. (Update - huge difference!)
Probably made a mistake with my yeast. I’d started up some Mangrove Jacks’ seltzer yeast to try… but then remembered how slow it is, so I pitched some fresh Kviek on top of it. Eek… I expect the Seltzer yeast to create some off flavours at 36°
Also made a mistake with the yeast nutrient - I completely forgot it until the fermenter was already in the fridge! Boiled some up and added it in, hopefully it doesn’t really need much boiling.
Result - this one wasn’t a winner for me. It was dry like I wanted, but the passionfruit was a big disappointment

Brew #6, 2021-12-28 - 2022-01-02: Plum, Pineapple, and Mandarin Seltzer

Recipe:

2kg Dextrose
Kveik Voss yeast
14g yeast nutrient
850g tinned Black Doris plums
420g tinned crushed pineapple
310g tinned mandarins

Notes:

Thoroughly crushed the plums and mandarins with the back of a fork this time
Had real trouble reinvigorating the yeast from the previous brew; might need to order a spare yeast for backup
10% Brix at start; finished just under 8% Brix… if this refractometer is accurate, these are very low ABV
Result - this was absolutely one of my favourites. Great outcome!

Brew #5, 2021-12-21 - 2021-12-25: Pineapple and Double Raspberry Seltzer

Recipe:

2kg Dextrose
Kveik Voss yeast
16g yeast nutrient
850g tinned Raspberries
420g tinned crushed pineapple

Notes:

Visually beautiful, great nose, and quite refreshingly delightful to drink.
The acidity of the raspberry and pineapple combined made this taste much like a kombucha. Weird.
Failed to take any gravity readings.

Brew #4, 2021-12-12 - 2021-12-17: Black Doris Plum and Pineapple Seltzer

Recipe:

2kg Dextrose
Kveik Voss yeast
12g yeast nutrient
850g tinned Black Doris plums
850g tinned pineapple (half crushed; half chunks)

Notes:

Initial tasting is very much drier than the others - not sure if this was the acids in the pineapple cutting the sugar, or if the sugar syrup in the cans was way thinner? Either way, this may end up a low percentage.
Original brix reading looked to be about 7%, suggesting an expected ABV of only 3.6%
(later): turns out I have no idea how refractometers work. Never mind.
This recipe came out amazing, absolutely delicious. Started out with a fairly subtle plum flavour, with the pineapple just becoming perceptible at the end.
However, during the week the plum flavour diminished and the pineapple flourished. This continued to the point where it was actually really unbalanced; I suspect it was caused by enzymes in the pineapple… to combat this, next time I’ll use only one tin.

Brew #3, 2021-12-04 - 2021-12-10: Lychee and Jackfruit Seltzer

Recipe:

1.2kg Dextrose, plus other assorted sugars to 2kg (approx 5.3%)
Kveik Voss yeast
16g yeast nutrient
850g tinned jackfruit
560g tinned lychee

Notes:

Brewing out much quicker now with the increased nutrient
Lychee is very prominent in finished product. Not overwhelming but definitely star of the show
Jackfruit brings very little flavour. Next time I might blend it up but worried about post-brew yeast collection

Brew #3, 2021-11-26 - 2021-12-03: Double Raspberry and Mandarin Seltzer

Recipe:

2kg Dextrose (for approx 5.3%)
Kveik Voss yeast
16g yeast nutrient
820g tinned raspberries
310g tinned mandarins

Notes:

Doubled the raspberry; reduced the dextrose; tripled the yeast nutrient
Raspberry far more prominent and yum. Zero impact from mandarin
Yeast nutrient really helped. Still a bit sweet but far better; probably no need to change
Adding the dextrose, plus approximate sugar syrup and fruit sugars, suggests a 6% ABV

Brew #2, 2021-11-17 to 2021-11-24: Raspberry and Mandarin Seltzer

Recipe:

2.5kg Dextrose (for approx 6%)
Kveik Voss yeast
5g yeast nutrient
410g tinned raspberries
310g tinned mandarins

Method:

Filled keg to 12l line with tap water
2.5kg Dextrose + 6g nutrient into 3 litres of water. Heated to 80°C
Took off heat, added tin of raspberries and tin of mandarins, including syrup; mashed up a bit
Brought temp back up to 75°C, and poured it into 16 litres of room temp water
Pitched Kveik yeast at 29°C. Total 20 litres ish, fermenting at 32°C

Notes:

Brew finished on 2021-11-24, exactly a week later
Kveik ferments much quicker, and very clean; happy
Mandarin very subtle, raspberry present but light. Still delicious
Didn’t add enough yeast nutrient!! Extremely sweet, but yeast has gone mostly dormant
Forgot to account for sugars from fruits; happy the yeast went to sleep before I got rocket fuel

Brew #1, 2021-10-25 to 2021-11-12: Mangrove Jack’s Pineapple Sunset Hard Seltzer

Brewed as per instructions, but added 12g of Mangrove Jack’s Passionfruit Beer flavour boost. Fermented a little slow; took a bit over two weeks to finish

Tasting notes:

Came out with the same acrid/chemical flavour that many others complain about
Brewing process super simple, though. Think I can do this myself without a kit

Acronis Cyber Protect and Microsoft Defender on the same Linux system

Sun, 07 Nov 2021 00:00:00 +1300

A few months ago, I published what I learned from playing with the Linux version of Microsoft Defender for Endpoint a few months back. If you’re here, you’ll have possibly already read my original post. There was a bit of tuning required to get it to behave nicely; just adding some sensible audit logging exclusions.

Well the Acronis Cyber Protect backup platform has recently been added to the bundle, and it had a few specific requirements, so I’ve documented them here.

There are four Acronis processes which MDATP will spend a lot of time scanning and rescanning. Exclude those from the scanning engine by executing the following commands as root:

mdatp exclusion process add --path /usr/lib/Acronis/Schedule/schedul2-bin
mdatp exclusion process add --path /usr/lib/Acronis/APL/active-protection
mdatp exclusion process add --path /opt/acronis/aakore
mdatp exclusion process add --path /usr/lib/Acronis/BackupAndRecovery/mms

There are also five processes which need to be excluded from auditctl, unless you want your audit logs rotated every ten minutes. Again, these must all be run as root.

echo -a never,exit -S 41 -S 42 -F comm="mms" -k exclude_acronis >> /etc/audit/rules.d/01-exclusion.rules
echo -a never,exit -S 41 -S 42 -S 43 -F comm="adp-agent" -k exclude_acronis >> /etc/audit/rules.d/01-exclusion.rules
echo -a never,exit -S 41 -S 42 -S 43 -S 288 -F exe=/usr/lib/Acronis/APL/active-protection -k exclude_acronis >> /etc/audit/rules.d/01-exclusion.rules
echo -a never,exit -S 41 -S 42 -S 43 -F exe=/opt/acronis/bin/updater -k exclude_acronis >> /etc/audit/rules.d/01-exclusion.rules
echo -a never,exit -S 41 -S 42 -S 288 -F exe=/opt/acronis/aakore -k exclude_acronis >> /etc/audit/rules.d/01-exclusion.rules

Once you’ve created the exclusions, you can either reboot or run service auditd restart. Either way, make sure you run service auditd status to ensure nothing went wrong with the exclusions.

Microsoft Defender for Endpoint (mdatp) on Debian Sid

Wed, 07 Jul 2021 00:00:00 +1200

** 2021-08-01 Update: **

Linux doesn’t have many great antivirus options available.

Don’t get me wrong, there are actually a few options nowadays. We’ve got the ever-present ClamAV; BitDefender has a good reputation, but I haven’t played with it yet; Sophos AV for Linux seems to be in limbo. ESET for Linux exists, as does Symantec Endpoint Protection for Linux. That’s a pretty decent range - but they each have some real drawback. Assuming the price is right, you still see compatibility, usability, licensing dramas, or resource drain challenges.

Most recently, the Linux version of Microsoft Defender for Endpoint entered the playing field. I finally got to play with this and I gotta say…

Holy shit.

Look, I know we enjoy ragging on Microsoft. They’ve demonstrated some awful behaviour over the years, and every action still harkens back to the “Embrace, Extend, Extinguish” modus operandi. But they’re doing really great work with the Defender/DATP/MDFE/whatever line. Even the basic Defender has been steading climbing the ranks, to the point where virtually all guidance I’ve heard for the last two years is: “if you’re not going full EDR, just use regulr Defender”. It’s no SentinelOne or CrowdStrike, but overall Defender just Gets The Job Done.

So, I was excited to try the Linux version. Long story short, it was a dream. The installation process was streaks ahead of the competition. Resource usage is generally negligible, adding an average load of around 0.5% CPU, going up to about 16% CPU during a scan. There’s a robust health and connectivity test built-in. Basically the whole thing just works, to the same level we’ve learned to expect from the Windows version. Oh and this one has actual documentation which seems to be considered optional by some of the competitors. 🙃

However.

You may find that mdatp is frantically scanning files that freqently Do Things. Bind9 (named) for example, might be handling hundreds of queries a minute. Each one of them triggers not only an mdatp scan, but also auditd entries… which often logs more than once per query. I was finding logs being filled and rotated every seven minutes in some cases… and then punctuated by auditd yelling “my buffer is full, I can’t log all these events!”.

So as a result, I’ve spent quite a few hours learning how to wrangle auditd around exclusions. Here’s what I’ve learned:

You need to lie to install MDFE on Sid

Edit /etc/os-release and change VERSION_ID="11" to VERSION_ID="10". After that, you can install MDFE following the regular instructions. Don’t forget to change it back later!

Troubleshooting performance issues:

First step - find the processes that mdatp is spending the most CPU time on:

# Download Microsoft's parser (only do this once)
wget -c https://raw.githubusercontent.com/microsoft/mdatp-xplat/master/linux/diagnostic/high_cpu_parser.py

# Ask Defender to output the stats, and then parse it looking for high CPU:
$ mdatp diagnostic real-time-protection-statistics --output json | python high_cpu_parser.py
925 	mongodb		29184
37575	pulseaudio	47
21355	packagekitd	30
37588	dbus-daemon	26
37550	sshd		25

… Yikes, it’s probably not necessary to scan MongoDB thirty thousand times! You can exclude a process from mdatp like this: sudo mdatp exclusion process add --path /usr/bin/mongod

But that’s not all… even though mdatp is no longer scanning the application, you still have auditd frantically logging every single thing it does. That’s also going to cause periodic slowdowns as the auditd buffer fills up, overflows, panicks, and then purges the stuff it couldn’t write down.

Syslog/audit logging issues

You can find out if anything is spewing a billion entries into your audit log, by just grepping for the rotation in syslog:

# grep 'Audit daemon rotating' /var/log/syslog
Jul 6 01:00:08 spam auditd[28996]: Audit daemon rotating log files
Jul 6 04:00:05 spam auditd[28996]: Audit daemon rotating log files
Jul 6 06:57:01 spam auditd[28996]: Audit daemon rotating log files
Jul 6 09:57:18 spam auditd[28996]: Audit daemon rotating log files

… Filling up a log file every three hours is weird, but previously this server was doing it every few minutes. You can see what’s causing all the log rotations with:

# cat /var/log/audit/audit.log* | cut -d ' ' -f26 | sort | uniq -c
 138581 
 1 exe="/bin/bash"
 4 exe="/bin/hostname"
 2 exe="/bin/nano"
 684 exe="/lib/systemd/systemd"
 4754 exe="/lib/systemd/systemd-logind"
 209 exe="/lib/systemd/systemd-user-runtime-dir"
 119 exe="/opt/microsoft/mdatp/sbin/osqueryi"
 9 exe="/usr/bin/apt-get"
 2133 exe="/usr/bin/dbus-daemon"
 13212 exe="/usr/bin/perl"
 10784 exe="/usr/bin/pmxcfs"
 14 exe="/usr/bin/rrdcached"
 2 exe="/usr/bin/sort"
 4 exe="/usr/lib/postfix/sbin/cleanup"
 880 exe="/usr/lib/postfix/sbin/pickup"
 175 exe="/usr/lib/postfix/sbin/qmgr"
 27 exe="/usr/lib/postfix/sbin/smtp"
 4 exe="/usr/lib/postfix/sbin/trivial-rewrite"
 16 exe="/usr/sbin/cron"
 10326 exe="/usr/sbin/ebtables-legacy"
 10326 exe="/usr/sbin/ebtables-legacy-restore"
 3 exe="/usr/sbin/postdrop"
 5 exe="/usr/sbin/qmeventd"
 2 exe="/usr/sbin/sendmail"
 56 exe="/usr/sbin/smartd"
 3255 exe="/usr/sbin/sshd"

The number on the left shows the number of times that binary has had it’s activity logged. This should clearly show the problem executables… today we’re looking at perl, pmxcfs, and ebtables-legacy. Once you’ve got that, you need to determine which syscalls are part of regular noise:

# grep /usr/bin/pmxcfs /var/log/audit/audit.log* | cut -d ' ' -f 4 | sort | uniq -c
 10856 
 8142 syscall=263
 1357 syscall=43
 1357 syscall=84

Referring to this document, we can see these syscalls are sys_accept, sys_rmdir, and sys_unlinkat. Since pmxcfs is the ProxmoxVS Cluster File System, these are frankly all pretty normal syscalls for it to make. Let’s ignore them.

echo -a never,exit -S 43 -S 84 -S 263 -F exe=/usr/bin/pmxcfs -k exclude_PVE_internals >> /etc/audit/rules.d/01-exclusion.rules

^ Gotta do that as root, btw, or just append the line however you see fit. Once you’ve added exclusions for all of your high-noise entires, run service auditd restart and then service auditd status to make sure it worked properly.

On my Zentyal box I had to add a bunch of entries to exclude logging of certain high-noise things. For example, DNS query resolution (via named) and Samba activity:

cat /etc/audit/rules.d/01-exclusion.rules

-a never,exit -S 41 -S 42 -S 49 -S 82 -S 288 -F exe=/usr/sbin/named -k exclude_DNS_queries
-a never,exit -S 43 -F exe=/usr/sbin/winbindd -k exclude_Samba
-a never,exit -S 41 -S 42 -S 43 -S 87 -F exe=/usr/sbin/smbd -k exclude_Samba
-a never,exit -S 41 -S 43 -S 87 -F exe=/usr/sbin/samba -k exclude_Samba

A word of warning: Don’t just ignore everything willy-nilly. The audit logging system exists so you can track activity on the system - blithely sending 100% of the log entries to the bin will undermine what the audit sytem exists for!

… That’s all I’ve got so far. I’ll keep updating this if I bump into anything new!

Recipe: Bangin' Satay

Mon, 24 May 2021 00:00:00 +1200

Had a hankering for a Satay Chicken tonight and kinda whipped something up based on a couple different things. This is more of a saucy satay, not a dry skewer-type. This takes a bit of juggling; I had three elements on the stove, and a few stages. Worth it, tho.

I’ve broken this into stages: the sauce, the veggies, and the noodles. For the benefit of shopping lists (and Whisk!) I’ve provided an awful monolithic pile of ingredients up-front. Yay.

Ingredients

2 large chicken breasts
2 Tbsp fish sauce
2 tsp ground Cumin seeds
2 tsp ground Coriander seeds
2 tsp Paprika
2 tsp Turmeric
3 tsp Curry powder
3 small chillies
2 onions
5 cloves garlic
1 thumb ginger
2 bay leaves or kaffir lime leaves
2 shoots lemongrass
1 400g can coconut cream
0.5 cup Peanut butter
1 capsicum
1.5c mung bean sprouts (for optional veggie side)
2 spring onions or garlic shoots (for optional veggie side)
1 capsicum, any colour (for optional veggie side)
2 Tbsp dried shrimp (for optional veggie side)
2 Tbsp dried shallots (for optional veggie side)
2 Tbsp peanuts, roughly chopped (for optional veggie side)
Soy sauce
Sesame oil
Egg noodles
Cayenne pepper or other chilli, to taste
Lime juice, to taste
Salt, to taste
Sugar, to taste
MSG, to taste (substitute chicken stock if you want)
Chicken stock, to taste
Vegetable oil

Method

Slice your chicken into reasonably chompy pieces. Maybe 1cm thick, and small enough to fit in your mouth.
Put the chicken in a bowl, pour in the fish sauce. Give it a stir, and leave it to soak.
Dice the onion, and get it in the frypan over medium heat, with only just-enough oil.
While the onion cooks, prepare your spices in a small bowl: cumin, coriander, paprika, turmeric, curry powder.
The onion is still cooking, so chop up your garlic, ginger, and chillies. Give the lemongrass a good thump too.
Get a second frypan onto your hottest element, and get it very hot. Start browning your chicken. Do this in HOT but small batches so it doesn’t steam.
Take the onion out of the first fry pan, and add another good tablespoon of oil.
When the oil is hot, tip in your bowl of spices. These are already ground, so they’re gonna burn fast - keep them moving, try not to let them smoke.
As soon as the spices are fried off (30~45 seconds), quickly throw your onion back in. Give it a stir, then tip in the garlic, ginger, and chilli. Crank the heat up!
Add peanut butter, peanuts, bay leaves, lemongrass, and the juice of half a lime. When the peanut butter is melted, add coconut cream and stir.
Start adding your chicken to the sauce whenever it’s ready. After all the chicken is added, reduce heat to a simmer. Try to remember to stir it sometimes.
Time to give it a taste! It’ll be bland - this is tuning time. I added a tablespoon of sugar, a teaspoon of MSG, a few grinds of salt, more lime juice, and a teaspoon of dry chicken stock.
Once you’ve struck the balance of sour/sweet/savoury/salty/spicy that you’re looking for, leave it simmer for another 20 minutes or so.
Prep some veggies for a side. I used mung beans, garlic shoots, and a capsicum. Chop these up.
In yEt AnOtHeR pOt, bring a bunch of water to a rolling boil (say 1.5 litres). Get your egg noodles in, and cook according to the packet (often 6mins).
Remember that frypan you fried the chicken in? Get that going again. When hot, throw in your side-veggies, along with some soy sauce and MSG.
Strain the water out of your egg noodles, and start them frying with some oil and soy sauce.
Toss the fried shrimp, fried shallots, and extra peanuts into your nearly-cooked side veggies. Give it a taste, add salt/lime juice/MSG to make it yummy.
QUICK, YOUR NOODLES ARE STICKING TO THE PAN! Turf those out into serving bowls. Spoon the tasty veggies onto half, and your chicken satay onto the other half.
Tell someone else they’re on kitchen cleanup duty, and sit down to enjoy!

Configuring a custom SSL certificate in Zentyal

Sun, 25 Apr 2021 00:00:00 +1200

Zuper-quick post for when this bites me again in the future!

Tonight I upgraded from Zentyal 6.2 to Zentyal 7.0. Smooth as butter, everything went great.

Until I logged in, and my shiny Actual Paid Money SSL certificate had vanished, replaced by a self-signed commoner’s certificate. Piffle.

Instructions for installing a custom SSL certificate in Zentyal are actually kinda shaky, so here you go:

cd /var/lib/zentyal/conf/ssl/
mkdir old
mv * old
nano ssl.key
nano ssl.cert
cat ssl.cert ssl.key > ssl.pem
chmod 0400 ssl*

That’s it! Your Private Key lives in ssl.key, and your SSL certificate lives in ssl.cert.

Reboot, and job done.

Blocking bad extensions and extortion with iRedMail

Mon, 12 Apr 2021 00:00:00 +1200

Turns out this is my third Amavis article. I guess it’s just one of those systems.

Heads up, if you’re trying to do this, make sure you also read my other article about the “banned_files_lover” thing.

Today I’m on a mission to:

Drop all incoming Office ‘97 files (they’re predominantly malicious these days)
Drop all incoming Macro-enabled Office 2007+ files (there aren’t legitimate reasons to receive these in my scenario)
Drop any emails containing a .onion address
Drop any emails containing a bitcoin wallet

Dropping Attachments

The attachment block is easily handled by Amavis. In iRedMail on Debian, the configuration file is found at /etc/amavis/conf.d/50-user. Open your config file, and scroll down to the section where the $banned_filename_re variable is set. Insert the following line:

 qr'.\.(doc|dot|docm|docb|xls|xlm|xlt|xlsm|xlsb|
 xla|xlam|ppt|pps|pptm|potm|ppam|ppsm|sldm)$'i, # Office '97-2003 and Macro-enabled files
 qr'.\.(adn|accdb|accdr|accdt|accda|mdw|accde|mam|maq|mar|mat|
 maf|laccdb|ade|adp|mdb|cdb|mda|mdn|mdt|mdf|mde|ldb)$'i, # Microsoft Access files
 qr'^\.pub$', # Microsoft Publisher files

That will block all core Office ‘97-2003 files, as well as all Macro-enabled Office 2007-365 files.

By default, this will silently reject mails containing these attachments. If you want senders to receive a bounce message, search for the $final_banned_destiny variable and make sure it’s set to D_BOUNCE.

Dropping .onion and BTC

This part happens in Postfix, and it’s more-super-easy than the last bit. To configure Postfix’s body checks, edit /etc/postfix/body_checks.pcre. I simply added the following:

/(\w+\.onion)/ REJECT This mail server does not accept references to .onion addresses (${1})
/(\b(bc1|[13])[a-zA-HJ-NP-Z0-9]{25,39})/ DISCARD Bitcoin wallet detected (${1})

I used the REJECT directive for testing, but to be honest I don’t want to send bounces back to these people, so I changed to DISCARD after it was working. You can see the difference above.

The text after the REJECT directive is returned to the sender, and the text after the DISCARD directive is logged. The variable ${1} contains the detected string, and is appended to the response message.

That’s the end! Safety. Yes.

Acknowledgements

Thanks to Brad and Hamish for this post - Brad for the regex, Hamish for the idea, and both of them for the review!

Using Caddy to enable MTA-STS

Tue, 19 Jan 2021 00:00:00 +1300

About 7 months ago, I left Nginx and moved to Caddy. I’ve found it super easy, and have now experienced using it as a reverse proxy, a static site server (this one!), as well as a hosting a handy place for me to copy my favourite Kaomoji.

Update 2021-01-20: I had a bug! By default, Caddy serves this without a Content-Type header. However, the RFC specifies that the response should be explicitly Content-Type "text/plain". Thanks to klausenbusk for letting me know!

Note: the Caddyfile fragment to generate https://shrug.ericlight.com looks like this:

shrug.ericlight.com { respond "<meta charset='UTF-8'>¯\_(ツ)_/¯" }

Anyway, once I’d used Caddy’s respond directive, it was a simple step to take it forward and use it to serve my handy-dandy MTA-STS policy file!

Simply add this new Server block to your Caddyfile:

mta-sts.ericlight.com {

respond "version: STSv1
mode: testing
mx: in1-smtp.messagingengine.com
mx: in2-smtp.messagingengine.com
max_age: 86401"

header Content-Type "text/plain; charset=utf-8"

}

This simple Server block will get you a shiny HTTPS certificate, redirect any HTTP visitors to HTTPS, and will provide enquirers with your MTA policy file. (Obviously, replace my domain name and MX entries with your own.)

Sharp observers will notice that the MTA-STS policy file should be hosted at https://mta-sts.ericlight.com/.well-known/mta-sts.txt… but that’s the elegance of the Respond directive in Caddy: wherever you go at mta-sts.ericlight.com, you get exactly that response!

Example: https://mta-sts.ericlight.com/my/hovercraft/is/full/of/eels.txt

Okay, that was easy… the next part is just creating your three DNS records:

Record #1: An A record for mta-sts.ericlight.com, pointing to your Caddy server.
Record #2: A TXT record at _mta-sts.ericlight.com, reading v=STSv1; id=20210112. (Note the underscore! Also, the ID can be anything.)
Record #3: A TXT record at _smtp._tls.ericlight.com, reading v=TLSRPTv1; rua=mailto:tls-reports@targetdomain.com. (update that email address)

… There you go! Now you have a shiny new MTA-STS policy, and your Reporting email target (configured in DNS Record #3) will start reciving daily reports from sending mail servers, letting you know if your policy is working correctly.

Eventually (once you’re sure it’s all configured properly), you should change your testing policy to enforce. When you do this, make sure you update the ID in your TXT Record! This is how sending mail servers know that there has been a policy update.

Enjoy!

Getting TrueNAS (FreeNAS 12) to work with an APC NMC2 UPS (AP9631)

Thu, 14 Jan 2021 00:00:00 +1300

TrueNAS, and FreeNAS before it, has supported UPS via the NUT software package since forever. But most people using it seem to be using USB-connected UPS devices. I don’t have one of these. I’m dealing with an IPv4-based APC ups, specifically the Smart-UPS X 1500 (SMX1500RMI2UNC), with an AP9631 NMC2 card.

You’re probably here because you’re in the same boat: trying to set up a non-USB or Ethernet-based UPS under FreeNAS/TrueNAS, and you can’t. You’ve noticed the TrueNAS console will not shut up about errors which read something like:

nut plugin: nut_connect: upscli_connect (localhost, 3493) failed: Connection failure: Connection refused. (Or perhaps port 161)

“But why?”, you say. “I’ve given TrueNAS my UPS hostname, why is it always trying to talk to localhost? And what is this port 3493, when SNMP is 161?”

What’s happening is that upscli is trying to talk to the upsd damon on your TrueNAS server, which is expected to listen on port 3493 - it’s upsd which then does the talky with the SNMP driver, which in turn does the talky with the UPS. This is why the TrueNAS console logs look like it’s just shouting at itself.

Okay, let’s get into it:

UPS Configuration

First, you need to enable SNMP on your AP9631. This was pretty straightforward, but remember that you need to go:

Configuration -> Network -> SNMPv1 -> Access -> Enable
Configuration -> Netowrk -> SNMPv1 -> Access Control, and make sure the public community is set to be accessible from 0.0.0.0
Reboot your NMC2 card.

This configuration should work with the defaults that the NUT SNMP driver expects - I suggest you get it working like this for now, then tweak to SNMPv3 once you have a confirmed-working connection.

TrueNAS Configuration

Your UPS configuration menu is under TrueNAS -> Services -> UPS. Configure your TrueNAS UPS section like so:

General Options section:

Identifier: Up to you, but do yourself a favour and make it ups.
UPS Mode: Master
Driver: “Various ups 3 (various) SNMP - RFC 1628 (snmp-ups, experimental)” – once you save, this will simply display as snmp-ups$(various)
Port or Hostname: The hostname of the ups, hopefully something simple like ups.mydomain.local

Monitor section:

Leave the Monitor User, Monitor Password, Extra users, and Remote Monitor all blank/default for now - you can change them once the UPS is talking to you. Note I don’t believe it’s important to change the Monitor Password under this configuration, as upsd only listens on the loopback address.

Shutdown and Email sections

I shall leave in your capable hands.

Other Options:

Auxiliary Parameters (ups.conf): This is where you override the SNMP defaults, if desired. By default, NUT will use SNMPv1 and the ‘public’ community. See here for all config options availble for the SNMP driver. You could do something like this, if you want:

community=yaysecurity
snmp_version=3
privProtocol=AES

Auxiliary Parameters (upsd.conf): LISTEN 127.0.0.1 3493

That last line was the magic - this is the bit that makes upsd not only listen on port 161 (whyyyy), but also listen on 3493, which is where the rest of the UPS subsystem expects to find UPS data!

Other interesting tidbits:

Running upsc -l should list all UPS instances configured on your server. Since I’ve got one, called ‘ups’, I see:

# upsc -l
ups

If upsc -l gives you a Connection Refused error, check /usr/local/etc/nut/upsd.conf to make sure it’s listening on 3493. If not, add your auxiliary parameter in TrueNAS and restart the service.

# upsc ups
Error: Connection failure: Connection refused

# cat /usr/local/etc/nut/upsd.conf
LISTEN 127.0.0.1 161
LISTEN ::1 161

# sockstat -4 -l | grep upsd
uucp upsd 99934 6 tcp4 127.0.0.1:161 *:*

Above: upsd is definitely not listening on the expected port

NUT Config files: These live under /usr/local/etc/nut/. The folder is deleted and recreated every time you edit something in the TrueNAS GUI, so if you’re in that folder and change something in the GUI, you’ll need to do the whole cd /usr/local/etc/nut again. No you can’t do cd ../nut, because the whole inode is gone, so the OS doesn’t know where you are anymore.
Testing your UPS: Once upsd is listening on 3493, you should be able to run upsc ups, and it will query your UPS for data. Note that “ups” is your UPS identifier from the “General Options” section, also found as the first line of /usr/local/etc/nut/ups.conf:

# cat /usr/local/etc/nut/ups.conf
[ups]
 driver = snmp-ups
 port = ups.mydomain.local
 desc = 
 pollfreq = 15

# upsc ups
ambient.humidity: 0.00
ambient.temperature: 23.0
battery.charge: 100.00
battery.date: 05/15/2017
battery.packs: 0.00
battery.runtime: 1221.00
battery.runtime.low: 600
battery.voltage: 54.50
device.mfr: APC
device.model: Smart-UPS X 1500
[...]

I think that pretty much covers it. I hope this has been helpful to someone - at very least, I hope it’s helpful to future-me!

It’s been a little while since I’ve worked with UPS management software. I like to think it’s because my day job isn’t in infrastructure anymore, but also to be honest the whole UPS scene is fairly ugly. I recall speaking with an electrical engineer from Schneider a while ago, who jokingly said something like “creating a new UPS or CNC communications protocol is almost a rite of passage for any new engineer”. I can see why… when you’re new, and see something superbly awful, it’s quite compelling to try and fix it! But decades now of dependencies really do add up to be a burden.

The Actual Best Sous Vide Eggs... ™

Mon, 28 Dec 2020 00:00:00 +1300

Inkbird were having a special in November, so I got a sous vide for Christmas! There are heaps of places with recipes for the best sous vide eggs, most notably the always-outstanding Serious Eats, but also an excellent write-up on Salt Pepper Skillet, and of course sous vide manufacturer Anova Culinary with a somewhat briefer post.

I submit to you that none of these are the best sous vide eggs. In my eyes, the best egg has a completely opaque white (not rubbery and not snot), with a semicooked gooey yolk like a cross between honey and custard. Like this one:

The closest recipe on the pages I cited is the 13.5 minute egg at 75°C. This is really close, but I found the yolk was a bit overcooked, and the white undercooked. Absolute conundrum.

My fix? A slightly hotter temperature, and take your eggs straight outta the fridge. I found 76°C (169°F) for the vaunted 13.5 minutes managed to get the outer white to a non-snot texture, while the pre-chilledness of the egg kept the yolk from overcooking.

Logging DNS queries, for both pfSense and Zentyal server

Sun, 06 Dec 2020 00:00:00 +1300

Logs of your client DNS queries can be a really good tool for incident response. I’ve finally got this implemented but it was much more of a struggle than I expected it to be, so here’s my story!

We’ve got a pfSense firewall running as a DNS fowarder, and a Zentyal server running BIND9 as the authoritative local server. The firewall rules block all UDP/53, so all DNS queries go either directly to the firewall, or (more commonly) to the Zentyal server for resolution. This means we have two places that resolve DNS queries which we want to forward off to our syslog server.

pfSense

Unbound is a super simple DNS forwarder, and the configuration is wonderfully straightforward. It did take me quite a while to figure it out, but in the end it was a simple oversight on my part.

To get pfSense/Unbound to forward DNS queries to your syslog server, simply open the Services -> DNS Resolver page, click ‘Display Custom options’, and add these two lines:

:::ini
server:
log-queries: yes

I spent hours fiddling before I realised I was missing the empty “server:” directive. -__-

This will log all DNS queries that Unbound deals with. These logs go to the standard internal pfSense log; if you want them to be replicated off to an external syslog server, go Status -> System Logs -> Settings. Scroll down to the “Remote Log Servers” section and add your syslog server there. (I’m not covering syslog listener config in this particualar post!)

Zentyal

Yeah this one was harder to figure out. Because Zentyal overwrites the config at every damn opportunity, you can’t just edit /etc/bind/named.conf.options, like what you find in most articles… it may work for a moment, but it’ll be overwritten.

The real answer to editing Zentyal’s config files is outlined at https://doc.zentyal.org/en/appendix-c.html#stubs:

:::bash
cp /usr/share/zentyal/stubs/dns/named.conf.options.mas /etc/zentyal/stubs/dns/named.conf.options.mas

Once you’ve copied the stub into a place where it won’t be clobbered, edit it and zoom right down to the bottom line, which looks like:

:::ini
logging { category lame-servers { null; }; };

This is the line you need to replace. Now, you can go as ham as you want here. I’ve actually used the sample provided by ISC themselves, which you can find at https://kb.isc.org/docs/aa-01526. However, that’s really robust and possibly more than you may need. A bare minimum would probably look like:

:::ini
logging { 
 channel queries_log {
 syslog named;
 print-time yes;
 print-category yes;
 print-severity yes;
 severity info;
 };
};

Note - when I tried to use the ISC sample wholesale, I needed to change a couple small things:

Changed all /var/named/log output destinations to /var/log/named
The lines for category zoneload and category trust-anchor-telemetry need to be removed for Zentyal 6.2’s version of BIND9
I needed to create /var/log/named, and make sure it was writeable by the bind user

I think that covers it, I hope this helps someone! I’ve actually even taken the time today to create a Pull Request in the Zentyal GitHub, we’ll see if it ends up getting merged!

Daaaaaal!

Tue, 24 Nov 2020 00:00:00 +1300

I’ve loved daal (or dal or dahl or dhal etc) since my trip to India in 2019… Ugh, that seems so interminably long ago now, in the final weeks of the long decade of 2020. Anyway.

So there’s this recipe I’ve followed a handful of times. I like it, it’s similar to what I enjoyed while travelling through Jaipur, but it doesn’t really kick the same way that the food in India did, so I always find myself tweaking it. The base recipe came from https://thewanderlustkitchen.com/indian-red-lentil-dal/ … and then I looked up a specifically Rajasthani daal recipe and found this one which is super similar. The place I’ve settled is in the middle of the two.

Ingredients and Method, jumbled into one

Step One (the pot)

3/4 cup Red Lentils - rinsed
1/4 cup Green Lentils - rinsed
3 cups water (not boiling, just room temp)

After rinsing, put the lentils and water in a medium pot over medium heat. Let them simmer while you prep the rest.

Step Two (the fry pan)

1.5 tsp Cumin seeds
1 Cinnamon stick
1 Tbsp butter or ghee
1 Tbsp oil

Put these into a medium fry pan; don’t turn it on yet.

Step Three (the base spices)

6 cloves of garlic, chopped
1.5 Tbsp of ginger, I just used the crushed stuff
1 large onion, diced. Use different types of onion if you’re feeling ✧･ﾟ: ✧･ﾟ: faaancy :･ﾟ✧:･ﾟ✧
2 green chilies. Dump the seeds and chop them up.
If you don’t have chilies, drop in 2 Tbsp of Salsa Picante instead, for that capsicum-type flavour.

Prep the above, and put them all into a small bowl.

Step Four (get your heat on)

At this point, turn the frypan on to medium heat. Let the cumin and cinnamon sizzle in the oil for about a minute, until the scent starts to pop.

Just before it starts to smoke, throw in the contents of your small bowl from Step Three, give it a stir, and crank the heat up to eleven.

Step Five (getting serious now)

1 tsp turmeric
1 tsp paprika
2/3 tsp cardamom (either ground, or smash the seeds in a pestle)
1/2 tsp red chilli powder
2 bay leaves
1 tomato, diced
Spinach! I used about a cup worth of the frozen chopped stuff (defrosted first, of course)

Give your frypan a stir every couple of minutes. By the time you’ve finished chopping the tomato and getting the rest of this together, the onions should be transparent and the whole thing should be smelling amazing.

Add all these ingredients to the frypan and give it another stir. Stir up the lentils as well, they should have been simmering for a few minutes by now!

Step Six (tune those flavours)

This last part really comes down to personal tastes, but here’s my go:

1/2 tsp salt
1/2 tsp Aromat or MSG
1 tsp Vegeta vegetable stock
1 tsp dried coriander
The juice of one lemon
1 cup of coconut cream, regular cream, or yoghurt

Once the lentils have thickened up, mix in all the above, and then stir in the now-cooked contents of your frypan. That’s it, your daal is assembled! Taste and tweak as desired, and then serve on rice.

Enjoy!

iRedMail, SpamAssassin, and Lynis

Mon, 02 Nov 2020 00:00:00 +1300

I really like iRedMail, and I also really like Lynis.

However, they don’t exactly like each other… or, more accurately, some of Lynis’ recommendations can cause a couple iRedMail components to fail. Today we’re talking about SpamAsassin.

One of the suggestions from Lynis is to turn off the ’execute’ bit on compilers for users who aren’t either the owner or in the owner group (the ‘other’ execute bit). For example:

:::bash
root@server:/# chmod o-x /usr/bin/as
root@server:/# chmod o-x /usr/bin/gcc

Easy peasy! But once you do this, you might start getting the following in your daily iRedMail Cron reports:

:::text
/etc/cron.daily/spamassassin:
/bin/sh: 1: x86_64-linux-gnu-gcc: Permission denied
make: *** [Makefile:346: body_0.o] Error 126
command 'make PREFIX=/tmp/.spamassassin23046Zmmrr9tmp/ignored INSTALLSITEARCH=/var/lib/spamassassin/compiled/5.028/3.004002 >>/tmp/.spamassassin23046Zmmrr9tmp/log' failed: exit 2

There are probably a bunch of ways to fix this. My way, I’m sure, is not the best way… however it was quick and easy, and it worked.

:::bash
root@server:/# chgrp debian-spamd /usr/bin/as
root@server:/# chgrp debian-spamd /usr/bin/gcc
root@server:/# runuser -l debian-spamd -c sa-compile

Look ma, no more errors!

This solution really only works for me because debian-spamd is the only non-root user that calls these compilers. If I had another user which needed to call them, I’d have to come up with a better fix. But for a standalone iRedMail server, this does the trick!

iRedMail: Daily user unknown entries from backup_sogo.sh

Sun, 01 Nov 2020 00:00:00 +1300

If you’ve been running iRedMail for a while, eventually you’ll probably start seeing ‘user unknown’ events in your daily logs:

:::text
* Backup all users' data under /var/vmail/backup/sogo/2020/11/01
<0x0x5574a64c36b0[SOGoCache]> Cache cleanup interval set every 300.000000 seconds
<0x0x5574a64c36b0[SOGoCache]> Using host(s) '127.0.0.1' as server(s)
2020-11-01 09:29:12.784 sogo-tool[29749:29749] user 'abdulm' unknown
2020-11-01 09:29:12.786 sogo-tool[29749:29749] user 'bent' unknown
2020-11-01 09:29:12.786 sogo-tool[29749:29749] user 'brettr' unknown
2020-11-01 09:29:12.786 sogo-tool[29749:29749] user 'catalinar' unknown
2020-11-01 09:29:12.787 sogo-tool[29749:29749] user 'clinth' unknown
2020-11-01 09:29:12.787 sogo-tool[29749:29749] user 'danield' unknown
2020-11-01 09:29:12.787 sogo-tool[29749:29749] user 'dannyn' unknown
2020-11-01 09:29:12.788 sogo-tool[29749:29749] user 'darcyk' unknown
2020-11-01 09:29:12.788 sogo-tool[29749:29749] user 'davidl' unknown
* Compress backup files.

This is because removal of an iRedMail user doesn’t remove the corresponding SOGo user data. You can take care of this with sogo-tool:

:::bash
root@server:/# sogo-tool remove abdulm@<domain.xyz> bent@<domain.xyz> brettr@<domain.xyz>

… etc. Once you’re done, run the backup again to make sure you’ve got them all:

:::bash
root@server:/# /bin/bash /var/vmail/backup/backup_sogo.sh
* Backup all users data under /var/vmail/backup/sogo/2020/11/01
<0x0x563fd0e2b6b0[SOGoCache]> Cache cleanup interval set every 300.000000 seconds
<0x0x563fd0e2b6b0[SOGoCache]> Using host(s) '127.0.0.1' as server(s)
* Compress backup files.

Your mate Eric, and his journey through depression

Thu, 08 Oct 2020 00:00:00 +1300

A couple of months ago, the team at my day job asked me to write something for Mental Health Awareness Week, so today I’m sharing some of my personal experiences around depression. It’s one of my longer articles, but the key message I want you to hear is: If you’re struggling, speak with someone. Seriously, just lean over to your mate now and say “Hey, did you see that article from Eric today?” – that’s all it takes to start a conversation, and believe me: sharing your feelings is a solid way to start getting on top of them.

The year was 2011, maybe February, about 10pm… the kids were in bed, school lunches packed, teeth brushed. It was a normal part of a normal bedtime routine for the whole household. I don’t particularly recall anything happening… but I remember looking at my wife and saying, “Shit… I think I’m depressed”. In the morning I phoned my doc (that was the hardest part) who started me on Fluoxetine - the medication that supported me throughout my recovery. Making that phone call the first step in a long journey, but it cemented my realisation that I was struggling, and that I needed help.

The New Zealand Mental Health Foundation says that one in six New Zealand adults have been diagnosed with a common mental disorder (depression, anxiety, bipolar) at some time in their lives. And behind every diagnosis, there are a host of other staunch people who have struggled and pushed to battle through it silently, sometimes with heart-breaking results. (Rest easy, Deano 💔). According to the World Health Organization (via health.govt.nz), mental illness accounts for 15% of the total burden of disease in the developed world, with depression set to become the second leading cause of disability in the world by *checks watch* … 2020.

Realising I was depressed came as quite a shock to me. I’ve always been the dude who has a smile on his face, laughs at every opportunity, and gets super enthusiastic about super niche things. I was running a reasonably successful business at the time, really enjoyed my work and my colleagues, had a thriving family… It didn’t make a whole lot of sense to me. Things were looking good!

In retrospect, all of that was true, but it’s only part of the picture. I was exhausted, working at least 60 hours a week, I rarely had real (uninterrupted) time to spend with my family, and there was a huge mental burden knowing that 16 individual lives were reliant on the income that the company created… a company which, though financially “fine”, was never financially “flourishing”. I’d lost touch with friends, stopped exercising, and wasn’t engaging in anything I truly loved. Things were not looking good.

Depression was not what I expected. People talk about the “black dog” of depression, and I guess I expected it to be “feeling sad all the time”. I rarely felt sad; I just felt numb and disengaged, and it felt like my head was full of mud. Instead of heading off on my mountain bike or making plans for after work, I’d rather sit on the couch and scroll through social media until bedtime arrived. But when I did end up in a social situation, I could easily engage and laugh and chat – enough that nobody suspected how I was feeling. Years later, when I was sharing my experiences with a friend, I described it as “the shine had gone off everything”. Disinterested, disengaged, foggy-brained, and no motivation to change any of it. (big note: this is how I felt it, but folks experience depression differently)

What helped me through? Firstly, medication. Antidepressants usually take a full two weeks to kick in, but before long I realised that the general foggy/muddiness of my brain was starting to lift. I could mentally take things in again, and I was a little less detached… enough that I could turn up somewhere voluntarily. Once I was able to actively engage a little, I started exercising a bit. We all know that exercise is a strong antidepressant, but I needed help in order to do that. And the third thing that helped – once I’d been on meds and exercising for a couple months, I had the capacity to visit friends, share stories, and make regular commitments. My mate Russell and I would catch up every Friday at 5pm, cook dinner for our combined families, and then spend hours talking, playing guitar, singing, and just generally enjoying friendship. That connectedness is what I finally needed to recover from depression.

Important note: Everyone is different, and that difference is rarely more pronounced than in mental health. For example, the medication that worked for me may not work for everyone, and medication often doesn’t work alone. Friends of mine have tried three or four different types of antidepressants before finding one that helped, and counselling often plays a critical role in recovery.

Connectedness with community and friends is one of our most powerful contributors to happiness and that sense of fulfilment we yearn for. Humans are very social creatures, and it’s essential to our wellbeing to engage and bring value to our social circle. We all need to put time into our friendships, our community, and our passions – neglecting these can bring a much greater cost than we expect, and it’s hard to dig yourself out once you’re in that hole. In fact – your friends and community are critical to pulling you out, which is why opening up and talking about it is so important. We can’t help if we don’t know.

To finish up this article, I thought I’d share a couple of resources. I went through my depression at the same time as Hyperbole-and-a-Half author Allie, who has a comical story here. A less-silly comic was drawn up by artist Colleen Butters here, which I also recommend. Lost Connections by Johann Hari is an excellent book, bringing together research from the last fifty years, with a focus on social connectivity. And if someone reaches out to you for support, there are some resources on how best to help at the Mental Health Foundation

Oh, getting back to the beginning of the story. Of course, my wife replied “Yeah, duh, I’ve been trying to tell you that for months.” ¯\_(ツ)\_/¯

Make the Flex theme center images

Wed, 07 Oct 2020 00:00:00 +1300

Previously I’ve been frustrated that the CSS in the Pelican Flex theme doesn’t centre images. However, I’m only one user with one use-case, so I haven’t raised a pull request.

Instead, I’ve finally gotten around to creating a quick-and-dirty sed string to fix the minified CSS file:

sed -i 's/}img{max-width:100%}/}img{max-width:100%;margin-left:auto;margin-right:auto;display:block}/' style.min.css

I also like my caption text to be italicised and centered, so I extended this a little:

sed -i 's/}img{max-width:100%}/}img{max-width:100%;margin-left:auto;margin-right:auto;display:block}figcaption{font-style:italic;text-align:center}' style.min.css

Run either of those in ./Flex/static/stylesheet. It’ll modify just the img class in the CSS and suddenly all your images will center unless you tell them not to!

Kate Bush - Army Dreamers guitar tab

Sun, 30 Aug 2020 00:00:00 +1200

Some friends and I have been working on our own version of Army Dreamers, by Kate Bush. The original version is replete with instruments, including a prominent Mandolin, some background violins, a synth of some sort, and even the sound of a rifle being racked. Since we’re two guitarists, a bassist, and a singer, there’s quite a bit of arranging to do to make it work.

Although there’s plenty of tab around for the mandolin part, that’s only a few notes, with the violin part kicking in immediately afterwards. I wanted to cover both.

There are four separate phases in it, which I’ve made up some unimaginative names for:

The slide phrase
The downwards walk
The upwards walk
The violin phrase

The bar structure runs like this:

Slide phrase + violin phrase
Downwards walk + violin phrase
Slide phrase + violin phrase
Upwards walk + violin phrase

That structure is then repeated for the entire song. As you can see, the violin phrase is actually dominant, being played at the end of every bar.

:::text
 The slide phrase The violin phrase Downwards walk phrase Violin phrase again
e|--17/14-14----17/9-9-----9h10p9------------------10----9----10--------------9h10p9----------------|
B|----------------------------------10--------------------------------7----------------10-----------|
G|---------------------------------------9--------------------------------------------------9-------|
D|-------------------------------------------9--------------------------------------------------9---|
A|--------------------------------------------------------------------------------------------------|
E|--------------------------------------------------------------------------------------------------|



 The slide phrase The violin phrase The upwards walk phrase Violin phrase again 
e|--17/14-14----17/9-9-----9h10p9----------------------------7----9-----------9h10p9----------------|
B|----------------------------------10-------------7----8------------------------------10-----------|
G|---------------------------------------9--------------------------------------------------9-------|
D|-------------------------------------------9--------------------------------------------------9---|
A|--------------------------------------------------------------------------------------------------|
E|--------------------------------------------------------------------------------------------------|

Ham and Pea Soup

Sun, 09 Aug 2020 00:00:00 +1200

I’ve never made Ham and Pea soup before today. I remember my mum making it when I was younger, but it was never really my thing. But tonight just seemed the right combination of a rainy, chilly, wintery evening… and we had some leftover jalapeño cornbread!

Well, it was so good I completely forgot to even take photos. I did it all in our pressure cooker, so the meal was ready in about half an hour. Here’s what I did:

Ingredients

Primary Aromatics

2 tablespoons butter
5 cloves garlic, chopped
1 onion, diced
1 shallot, diced (or just use a bigger onion)

Secondary Aromatics

2 carrots, chopped about 1cm square ish
2 celery sticks, chopped about 1cm square ish
2 bay leaves
decent pinch of dried parsley

Body

a bacon hock, circa 800g
500g bag of dried split peas
1/2c red lentils
2.5tsp Vegeta vegetable stock powder
about 6 cups boiling water

Method

Start by putting the peas in the pressure cooker pot, and soaking them in fairly hot water. Then start your prep.
Once the veggies are prepped, drain the peas and set them aside. Rinse the pot, and put it on the stovetop on high.
Add your primary aromatics and fry for 5 minutes, then add the secondary aromatics and fry another 10 minutes.
Remove from stovetop, and put the inner into the pressure cooker body.
Add the peas and lentils to the pot, sprinkle in the vegetable stock, and mix it all together.
Create a depression in the mix, and place the bacon hock in, so it’s as low in the pot as possible.
Fill with boiling water until covered, and try to stir it a bit without lifting the hock too much.
Put on lid, and pressure cook for 30 minutes.

That’s about it! After that it was just a matter of chopping the meat off the hock, stirring it in, and potentially adding a little more water.

It would have been far easier to use a boneless lump of ham instead of the boney bacon hock. If you do this, add some extra stock powder to compensate for the missing salt.

Credit: The recipe above is based on https://www.allrecipes.com/recipe/219170/ham-and-split-pea-soup-recipe-a-great-soup/, with a good few tweaks of my own.

Delicious Cornbread

Sun, 26 Jul 2020 00:00:00 +1200

I remember celebrating Christmas with family in New Mexico when I was a kid. I was fairly young, but I think it must have been at the Feast of the Virgin of Guadalupe. One of the most delicious treats I recall from the Feast was real Mexican cornbread.

Now, every few years I’ll head into Gilmours and yoink a three-kilo bag of cornmeal. I don’t make this often, but we do all love it, and one batch usually makes enough for multiple households.

I always cook this in a cast-iron skillet, because it allows me to fry off the butter a bit first - this adds a nice roasty nutty flavour.

Ingredients

Delicious fat

120g (half a cup) of butter

Dry ingredients

1.5 cups cornmeal
1.5 cups other flour (mix & match anything to add up to 1.5c; just avoid self-raising)
2.5 tsp baking powder
2 tsp salt
1 cup brown sugar

The ‘other flour’ component is super flexible - as long as it’s dry dust, it hardly seems to matter what you use. Rice flour could make this gluten free, or wholemeal to make it gut-friendly, or tapoica flour if you just … need to use up some random tapioca flour you bought over lockdown for some reason.

In fact, the cornbread in the oven right now has some psyllium husk, to try and keep the moisture locked in. (Update: the psyllium was awesome! Substituted 1Tbsp for random flour.)

Wet ingredients

3 eggs, room temperature
1.5 cups milk, room temperature

Method

You forgot to warm up your eggs and milk, so pop your milk into the microwave for a minute, and stick the eggs in a bowl of warm water.
Turn your oven to 180°C, then pop the butter in your skillet and get it melting over low heat.
Mix your dry ingredients in a mixing bowl. In different (larger) mixing bowl, whisk together the eggs and milk.
Add the dry ingredients to the wet ones, folding gently. Now you can just dust the flour out of that bowl and put it away! (Life haxx)
Your butter should be a nice nutty brown by now; carefully pour it into your batter - keep just a tiny bit in the pan for non-stickness.
Give everything a gentle stir, and pour the batter into the skillet.
Into the oven it goes! Cook until a toothpick comes out clean, approx 20~30 minutes.

Jalapeño Update

Today I tried some jalapeño and cheese in the cornbread. It was easily the best cornbread I’ve made so far!

As above, except with the following modifications:

Increase salt to 2.5tsp
Reduce sugar to a half cup
2~2.5 cups of grated cheese
About 1/3 cup of chopped jalapeño

The cheese and jalapeño go into the batter after the butter is mixed in - just before you pour it into the skillet.

Today’s cheese was about 1.5 cups of grated Colby, and another half cup of grated parmesan to add a bit of sharpness. It wasn’t quite enough, so next time I’ll either increase the proprtion of parmesan to 2/3c, or use a slightly sharper base cheese.

Credit: The recipe above is almost entirely from https://cookieandkate.com/honey-butter-cornbread-recipe/, however I’ve made a few small tweaks.

I made Salmon Gravlax!

Sat, 11 Jul 2020 00:00:00 +1200

I joined some friends last weekend for a belated World Whisky Day celebration. The entry fee was, of course, a bottle of whisky (this year the theme was different finishing casks)… but we also each bring some delicious morsel that piques our interest.

This year, as I was bringing Glenmorangie’s The Lasanta (aged in Oloroso and Pedro Ximénez sherry casks), I decided my amuse-bouche would be sherry-based.

My treat was dried prunes, soaked in a dry sherry, then sprinkled with a pinch of salt, and eaten with a tamari-roasted almond. They were quite delicious - you only want to soak them for 5 hours or so though, they do go mushy. The salt is critically important; I’m not sure about the tamari, plain almonds may have been just as good.

Anyway, although my treat was great, it paled in comparison to one of the other treats - an amazing fresh salmon gravlax. I’d never heard of gravlax before, but it’s actually just a method of curing salmon - sit it in a casing of sugar, salt, and dill for a couple days and it’s done! Of course, I had to try it.

The recipe below has a couple tweaks, and it’s taken almost entirely from https://www.recipetineats.com/cured-salmon-gravlax/.

Comments

For the salmon, skin on or off doesn’t matter. I did mine skin-on, and it’s a little annoying - next time I’ll take it off.
The larger the salt granules, the better. I used this Himalayan rock salt. Avoid iodised salt, apparently.
Prefer bigger sugar granules, too - probably not coffee crystals, but raw sugar worked really nicely. Avoid icing sugar or caster sugar.
Use peppercorns if you can; they’re fresher than pre-ground. If using pre-ground, add a bit extra. I used these ones.
Curing time is tricky. I did mine for 86 hours and it was definitely a hard-cure, but was still really good. I imagine the whisky weekend version had only been a 48-hour cure.

Ingredients

Salmon (de-boned)
Salt & Sugar (ratio below)
Pepper (about 2tsp per 500g salmon)
Dill (optional - about 25g per 500g salmon)
Coriander seeds (optional - about 2tsp per 500g salmon)
Grated lemon rind (optional - about a lemon’s worth per 500g salmon)

Method

Bruise your dill, crush your peppercorns, whack your coriander seeds, and grate your lemon rind
Mix your cure. I used a recipe of 4:1:1 (fish:sugar:salt), but next time I’m doing 5:1:1.
Once your cure is mixed, fold in your spices and aromatics.
Spread out half of your cure, lay the salmon on top, and then spread the rest of the cure on top.
Bag it up! I used a vacuum sealer, but a sandwich bag or some plastic wrap will do the trick.
Pop it in the fridge with a weight on it (people recommend a plate).
Flip it over every 12-24 hours. Be careful - a lot of liquid will come out during the curing process!
Once you’ve cured it for 24-72+ hours, pull it out and gently rinse off all the clumps of dill/pepper/salt etc.
Wrap it in a paper towel to dry it off, then leave it unwrapped in the fridge for a few hours to finish drying out.
Enjoy! Slice it with a nice sharp knife. I’m told it should be sliced “thin enough to read a newspaper through”.

Other thoughts

I’ve seen recipes that use aquavit… So next time I’m actually going to try a few tablespoons of gin!
The 4:1:1 ratio wasn’t bad - it’s just that I ended up pouring out lots of big chunks of salt, so I think it was unnecessary wastage.
I’ve didn’t try coriander in this batch, but I’m super-keen to try it in the next one. I’ve also seen recipes use juniper berries, which is what made me think of the gin.
What to eat with gravlax? We had it on a platter that also contained seed crackers, a couple cheeses, fruit paste, salami, capers, and pickled gherkins. The capers were great.

My mountain

Sat, 06 Jun 2020 00:00:00 +1200

I got a tattoo a couple years ago, which is a sketch outline of Mount Tongariro and Mount Ngauruhoe, in the Central Plateau of New Zealand.

The photo that the tattoo, which the tattoo was based on, was just the result of a Google search. I’ve tried to find it many times since I actually got the tattoo - at very least just so I could give credit to the photographer. However, to no avail.

The other day I happily found a copy of the photo, and went to Google’s Reverse Image Search to figure out where it had come from! The answer is…

… nowhere. Whoever posted the image way-back-when appears to have taken the photo down. The Mighty Goog can’t find it anywhere.

So, here we are I guess. I’m uploading the photo here so I don’t lose it again. If you took this photo, or if you know who did, please contact me!

ZFS zpool vanishing after upgrading ProxmoxVE 5.4 to 6.2

Sat, 23 May 2020 00:00:00 +1200

Performing a major version upgrade is never pleasant. I’ve been using ProxmoxVE for about ten years now though, and it’s consistently done a fantastic job. Since it’s based on Debian, all the upgrades are done with a simple apt update && apt upgrade, with a variety of steps in the middle to point to new repositories, etc. Nothing out of the ordinary, nothing scary.

So I’ve got two servers - one is an old one, with only a couple testing VM’s residing on it. I go ahead and work through the 5.x to 6.x upgrade guide, and everything Just Works. No problem with that one.

When I moved onto the other server I had a couple different things I noticed. For example, when trying to upgrade Corosync 2.x to 3.x (a prerequisite for the OS upgrade), I noticed it was trying to uninstall things like corosync-pve, which really didn’t seem right.

After comparing dpkg -l contents between the two servers, I learned that corosync-pve is actually just a transitional package - completely replaced by the new corosync package in version 3. And although the server was trying to uninstall corosync-pve, it was then immediately installing corosync… so no problem.

Things continued to be mildly alarming during the upgrade itself, but not actually toooo bad, as all the removed packages were being replaced. This one, for example:

dpkg: pve-libspice-server1: dependency problems, but removing anyway as you requested:
 pve-qemu-kvm depends on pve-libspice-server1 (>= 0.12.5-1); however:
 Package pve-libspice-server1 is to be removed.
 spiceterm depends on libspice-server1 (>= 0.12.2); however:
 Package libspice-server1 is not installed.
 Package pve-libspice-server1 which provides libspice-server1 is to be removed.

(Reading database ... 65255 files and directories currently installed.)
Removing pve-libspice-server1 (0.14.1-2) ...

… was replaced with spiceterm. No worries.

However things went sharply downhill following reboot. All of my VM images were missing! I had a ZFS zpool that should have been at /VM_Local_zpool - that folder existed, and contained an ./images folder, but that was completely empty.

I quickly checked the storage information in Proxmox, and the result was… not good:

… Really, not good. :-|

So I SSH’ed into the server and had a quick look at my list of ZFS stores, and discovered that my dataset was entirely absent:

# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 83.8G 23.8G 96K /rpool
rpool/ROOT 3.65G 23.8G 96K /rpool/ROOT
rpool/ROOT/pve-1 3.65G 23.8G 3.65G /
rpool/data 71.6G 23.8G 71.6G /rpool/data
rpool/swap 8.50G 25.4G 6.85G -

# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 111G 82.1G 28.9G - - 66% 73% 1.00x ONLINE -

A bit of Googling led me to the zfs import command, which resulted in the first piece of good news of the night:

# zpool import

 pool: VM_Local_zpool
 id: 6614373363984244305
 state: ONLINE
 status: Some supported features are not enabled on the pool.
 action: The pool can be imported using its name or numeric identifier, though
	some features will not be available without an explicit 'zpool upgrade'.
 config:

	VM_Local_zpool ONLINE
	 mirror-0 ONLINE
	 wwn-0x50014ee0aeee96ef ONLINE
	 wwn-0x50014ee05998ee25 ONLINE

It exists! And it’s … online? But where? I don’t get it. I went ahead and tried importing it, but got a “directory not empty” error:

# zpool import VM_Local_zpool 
cannot mount '/VM_Local_zpool': directory is not empty

Now, if I’d been paying attention there, I would have realised that zpool list now contained my VM_Local_zpool:

# zfs list
NAME USED AVAIL REFER MOUNTPOINT
VM_Local_zpool 1.16T 611G 1.16T /VM_Local_zpool
rpool 83.8G 23.8G 96K /rpool
rpool/ROOT 3.65G 23.8G 96K /rpool/ROOT
rpool/ROOT/pve-1 3.65G 23.8G 3.65G /
rpool/data 71.6G 23.8G 71.6G /rpool/data
rpool/swap 8.50G 25.4G 6.85G -

From this point, all I had to do was empty /VM_Local_zpool, and try to remount the zpool. But there’s a second trick here.

ProxmoxVE has a pretty clear idea of How Storage Should Look, so every time it scans a storage device, is makes sure there’s an ./images folder in any storage enabled for VM Images. It also makes sure there’s a ./dump folder on any storage enabled as a backup target, and a ./template folder on any storage enabled for ISO images. (Reference: https://pve.proxmox.com/wiki/Storage:_Directory)

So, when I removed that /VM_Local_zpool/images folder, Proxmox just… went ahead and recreated it almost immediately, so I still couldn’t mount the zpool there.

In the end, I managed to get my zpool remounted by pairing the rmdir and mount commands together on the same line:

# rmdir /VM_Local_zpool/images/
# zfs mount VM_Local_zpool
cannot mount '/VM_Local_zpool': directory is not empty
# rmdir /VM_Local_zpool/images/ && zfs mount VM_Local_zpool
# cd VM_Local_zpool/
/VM_Local_zpool# ls
backup	images

… and there we go! My mount point, and all my VM images, are back unscathed!

And just for future reference - I saw a post on the FreeNAS forums that I should the status of the zfs-import-cache service, and enable it if it wasn’t started by default… however it was fine:

# systemctl status zfs-import-cache.service
● zfs-import-cache.service - Import ZFS pools by cache file
 Loaded: loaded (/lib/systemd/system/zfs-import-cache.service; enabled; vendor preset: enabled)
 Active: active (exited) since Sat 2020-05-23 20:46:54 NZST; 3min 26s ago
 Docs: man:zpool(8)
 Process: 1819 ExecStart=/sbin/zpool import -c /etc/zfs/zpool.cache -aN (code=exited, status=0/SUCCESS)
 Main PID: 1819 (code=exited, status=0/SUCCESS)

May 23 20:46:54 ~ systemd[1]: Starting Import ZFS pools by cache file...
May 23 20:46:54 ~ zpool[1819]: no pools available to import
May 23 20:46:54 ~ systemd[1]: Started Import ZFS pools by cache file.

Why are all Split-Tunnel VPN diagrams so ugly?

Thu, 21 May 2020 00:00:00 +1200

Right.

Today I needed to find a diagram of the traffic flow for a split-tunnel VPN. Nothing fancy, just a real simple user-facing diagram to form part of an article.

And friends, there was nothing. Nothing at all. I found proper technical ones from Cisco Meraki; fancy Office365 ones, depicting an ExpressRoute to O365 and tunnelling everything else through the VPN; and another Office365 tunnel with ExpressRoute and a split-tunnel for the rest of the traffic.

Pretty much the closest I came to my needs was this atrocity, via http://blog.soundtraining.net/2013/03/how-to-configure-split-tunneling-on.html:

So I went over to https://www.draw.io and whipped up something a bit prettier. This isn’t marvelous, but it’s way better than I could find anywhere else:

In retrospect, I could have done better - I really should have had the corporate tunnel going through the cloud of the internet. However this works for ne.

If you’d like to use this, feel free. You can even use this fancy URL to open the drawing as an editable vector graphic at Draw.io directly!

Roundcube not loading email contents

Mon, 11 May 2020 00:00:00 +1200

Here’s another quick one! I had a Roundcube instance, chugging along fine.

Then one day it just stopped loading emails! Everything else was all fine - no problems with login etc, but trying to preview an email just game me a sadfaec. Chrome couldn’t load the contents, and told me that the server had refused to connect:

So of course, I can’t just let this go - being able to see the contents of the emails is relatively important! But what the heck was causing it?

I spent some time in Chrome’s developer tools and console, as well as time grepping and tailing log files… this would have all been fixed much earlier if I were running Firefox I think, as it’s a bit more explicit with errors.

So, what was the fix? It was nothing to do with Roundcube, routing, firewalls, or reverse proxies. (Seriously, I checked all of these…)

The problem is that, last week, I’d modified the HTTP Headers on the reverse proxy to include:

 X-Frame-Options ("DENY")

… no good. It all came right when I changed the X-Frame-Options directive to “SAMEORIGIN”. Suddenly, all fixed.

Sheesh!

Renaming a computer object in Active Directory

Sat, 09 May 2020 00:00:00 +1200

Quick post tonight.

You can’t rename a computer object in Active Directory Users and Computers. Even if you change the hostname, the domain object will still have the original distinguishedName from the old device.

However… you can do it in ADSIEdit! Just open ADSIEdit, navigate to the AD container that holds your misnamed object, right click on it, and Rename.

I’ve just done it with a couple test computers on a domain and both worked completely smoothly. I don’t know if this would be safe if the renamed computer were off-site.

WireGuard on Windows - Part 2

Fri, 08 May 2020 00:00:00 +1200

A few days ago I spun up a Windows Dev VM to have a play with WireGuard for Windows.

I didn’t really have a clear goal in mind when I started playing with this - part of it was in trying to create and launch a tunnel without using the GUI. Mostly I was just trying to learn more about a new implementation of a tool that I really like.

If you read my previous article, you’ll recall that I started off trying to do this with just wireguard.exe. Here are some things I’ve discovered:

wireguard.exe does a whole lot of stuff

Wireguard.exe isn’t just a GUI, which I originally thought it was. It’s also the piece of software that shouts out to WinTun to create the interface, as well as the utility that reads the ’extended’ attributes in your .conf file (e.g. the stuff that wg-quick takes care of), as well as the utility that sets up your routes, DNS, etc, etc, etc.

wireguard.exe doesn’t create private/public keypairs

… To do this, you instead need to use wg.exe, which is installed under your System32 folder (so it’s in your path, so it’s accessible anywhere):

PS C:\> wg genkey | tee $ENV:APPDATA\WireGuard.priv | wg pubkey > $ENV:APPDATA\WireGuard.pub

I did figure this out the other day, but I’m reiterating here.

wg.exe can read .conf files – but you don’t really want it to

Last time, I was having trouble reading a .conf file from wg.exe:

PS C:\Users> wg setconf wg0 .\wg0.conf
Line unrecognized: ` ■['
Configuration parsing error

I’m not entirely sure what this was - I’ve since been able to read in a .conf file perfectly fine (for the record this is UTF-8 with Windows CRLF line-endings).

However … my breakthrough kinda sucks.

Wireguard for Windows stores it’s config files in the Windows DPAPI-encrypted vault. This is vastly better than just bunging a file in C:\Users\Blah and hoping for the best. Maybe it’s not perfect - I don’t know much about DPAPI - but it’s a far cry better than nothing.

When you use wireguard.exe to import a tunnel from a .conf file, it will read it in, sanity-check it (mine failed because I accidentally hit the keyboard during copy/pasta, so it rejected the Base64 encoding), and then safely store it away in the DPAPI storage. You can then delete your original .conf file. Just do this, it’s better.

You need both wireguard.exe and wg.exe

OK so here’s the bit that I only fully realised tonight: wireguard.exe is like wg-quick, but it also provides the interface into the Windows network stack and the Windows DPAPI storage of your sensitive conf files. You can’t even run wg set without it, because wireguard.exe is even responsible for creating the IPC Server that interfaces with WinTun.

Honestly the Windows world is so much more complicated than the Linux world*. 🙄

(* some caveats apply)

Conclusion

What’s my plan now?

I’m going to make a thing that:

Installs wireguard.msi silently
Uses wg.exe to create a private/public keypair
Uses that keypair to create a temporary .conf file
Uses wireguard.exe to import that .conf file from some predetermined location
Deletes the .conf file (probably using cipher.exe to scrub it)
Uses wireguard.exe to install the tunnel so it opens automatically on login

I’m hoping that this will create us something like an always-on-VPN connection that can be deployed easily by an IT support person, and require no end-user interaction.

Getting WireGuard on Windows - quietly

Mon, 27 Apr 2020 00:00:00 +1200

So, I mentioned in my post yesterday that I’m trying to get a bit of a quiet installer for WireGuard on Windows. Not that the current one is noisy, but I have a really simple use-case that I want to meet.

I spent some time seeing if I could just extract WireGuard.exe and run that. Nope, of course - it requires WinTun to be installed to facilitate the Layer 3 tunneling.

Next I played with the idea of just deploying WinTun directly, without installing the WireGuard package itself. I can only assume that this would have been super easy if I’d ever built a proper MSI installer before, but in the end it was just way easier to:

WireGuard-amd64-0.1.0.msi /q

… It was still worth the tangential digging, but who would have ever guessed that Jason Donenfeld would have already done things the best way. ¯\_(ツ)_/¯ </s>

Note you’ve got to run the above in an elevated command prompt, since it’s installing drivers. However, the installation is still not quiet - after install, it pops up the WireGuard GUI window!

“That’s annoying,” I think to myself, “can I stop that?” To the installer source code!

OwO … Hwat’s this?

 <!--
 Launch WireGuard.exe after setup complete
 -->
 <CustomAction Id="LaunchApplication" HideTarget="yes" Impersonate="no" Execute="deferred" FileKey="WireGuard.exe" ExeCommand="" Return="asyncNoWait" />
 <InstallExecuteSequence>
 <Custom Action="LaunchApplication" Before="InstallFinalize">(&amp;WireGuardFeature = 3) AND NOT DO_NOT_LAUNCH</Custom>
 </InstallExecuteSequence>

It appears that we can pass through an environment variable to stop the installer from launching the GUI after install!

 .\WireGuard-amd64-0.1.0.msi /q DO_NOT_LAUNCH=True

Aaaand success… now I’ve got WireGuard and WinTun installed, no user interaction, so far completely silent. Next I gotta create a keypair. There aren’t any relevant command-line arguments that I can pass to wireguard.exe, so I went on a hunt for a keypair generator.

I spent an hour or so scratching around various options to find a keypair generator that looked trustworthy. I honestly couldn’t find one… even the one posted on the WireGuard GitHub includes a caveat from Jason.

Until I decided to have a closer look at the installer source code… the very same code I was looking at when I discovered the DO_NOT_LAUNCH property above.

<Component Directory="System64Folder" Win64="yes" Id="Wg64Executable" Guid="d9b494ec-0959-442c-89ad-6aa175acfd03">
 <File Source="..\$(var.WIREGUARD_PLATFORM)\wg.exe" Id="Wg64Executable" />
</Component>

So C:\Program Files\WireGuard\WireGuard.exe appears to be just a shell that calls out to C:\Windows\System32\wg.exe. Oh my god that knowledge would have saved me hours!!

So… here’s how to generate a WireGuard keypair, in Windows, without the GUI. Since system32 is in the path, you can run this from anywhere. The below will create WireGuard.priv and WireGuard.pub in your AppData folder:

PS C:\> wg genkey | tee $ENV:APPDATA\WireGuard.priv | wg pubkey > $ENV:APPDATA\WireGuard.pub
PS C:\> cat $ENV:APPDATA\WireGuard.*
uIGDK+LMDZANniFxrofIpu/hUyezuwCM7qSDVVO+1Gw=
85MxGcWutHJsjFggiZ+J4/vsNYDVEa8zYk53DQZCwyE=

AWESOME. Next step is to zoink that into a handy-dandy config file. (More accurately, next step is to delete that private key that I’ve now posted to the internets)

Right, so it looks like I can’t just create a text-based wg0.conf file like I do in Linux. When I do, I get this error:

PS C:\Users> wg setconf wg0 .\wg0.conf
Line unrecognized: ` ■['
Configuration parsing error

I thought this might be the file encoding, so I tried it in ANSI etc, but then I learned that Wireguard for Windows stores it’s config files in C:\Windows\System32\config\systemprofile\AppData\Local\WireGuard\Configurations\ … and they’re not in plain text. They’re stored with the Windows Data Protection API (DPAPI), so perhaps that’s what wireguard.exe is expecting?

Either way, I’ve been playing with this for quite a long time now, so I’m off to take a break for the night or so. Once I figure out how to create a WireGuard config file for an interface I’ll post a new article. Hope you enjoyed reading so far!

A pork rub for when you've run out of garlic

Sun, 26 Apr 2020 00:00:00 +1200

I picked up a set of whole pork loins at Pak ’n Save the other day for like twelve bucks, and being on dinner duty tonight I thought it was time to fire up the smoker. But just last night we’d had a Chicken Tikka Masala which cleaned out our garlic stocks … what to do?

After a bit of dithering, I made a simple low-garlic rub and it turned out good enough that I’m documenting it here!

Outta Garlic Pork Rub

1 tbsp salt
2 tbsp brown sugar
1 tbsp Cajun seasoning mix (this had garlic in it)
1 tbsp ground cumin
1 tbsp ground coriander
1 tbsp smoked paprika
1/2 tbsp ground ginger
1/2 tbsp chilli flakes
1/2 tbsp turmeric

I covered a plate in the rub, then on a different board I sprinkled rub all over the tops of the pork, placed them rub-side-down on the plate-o-rub, and covered the remaining surfaces. I still have probably 2 tbsp of rub left, so I must have used 7 tbsp in total.

Into the fridge for about two hours, and then onto the smoker! I did it over pecan wood until it reached about 60°C, and then boated it in a 50/50 mix of apple juice and ginger beer until it hit an internal temp of 71°C. Then I let it sit for about 30 minutes before we sat down to dinner!

Reflections:

I was in a rush this time - no low & slow here. I really want to try it again over much lower heat.
I think it was a bit overcooked… 71°C is a bit hot for pork. Next time I’ll pull it off at 65°C and let it sit for a bit longer.

Overall, it still got a great score from the rest of the fam (in fact, they’re the reason I’ve recorded this recipe here)!

Setting up a Windows Dev VM under ProxmoxVE

Sun, 26 Apr 2020 00:00:00 +1200

Righto, I’ve been using WireGuard for quite a long time now, but I just had my first foray into WireGuard on Windows. I’m trying to roll it into a low-touch VPN deployment sort of thing.

There will be a post about that soon, but this particular post is about setting up the environment for me to play with this stuff… I need a system where I can take snapshots and roll back the drive to a known state. (So I can be sure I’m not missing anything that the installer would do).

I don’t have a Windows machine, so I started off by downloading a Win10 Dev VM from Microsoft. Since I’m running ProxmoxVE, I chose the VMware image, which will come down as a .zip file.

Download the zip file, and extract it to wherever your VM disk images live (mine is /VM_Local_zpool/)… this will save some time when you’re importing the disk.

Next, you simply use the importovf function of qm to import the OVF manifest as a new KVM-based VM in Proxmox!

~ # qm importovf 103 WinDev2004Eval.ovf VM_Images --format qcow2 Formatting '/VM_Local_zpool/images/103/vm-103-disk-0.qcow2', fmt=qcow2 size=136365211648 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16 (100.00/100%)
~ #

That was suspiciously easy…

After importing I couldn’t launch the VM, but I’ve seen this error a dozen times and know who to handle it:

kvm: -drive file=/VM_Local_zpool/images/103/vm-103-disk-0.qcow2,if=none,id=drive-sata0,format=qcow2,cache=none,aio=native,detect-zeroes=on: file system may not support O_DIRECT kvm: -drive file=/VM_Local_zpool/images/103/vm-103-disk-0.qcow2,if=none,id=drive-sata0,format=qcow2,cache=none,aio=native,detect-zeroes=on: Could not open '/VM_Local_zpool/images/103/vm-103-disk-0.qcow2': Invalid argument TASK ERROR: start failed: command '/usr/bin/kvm <-- snip --> -machine 'type=pc'' failed: exit code 1

If you’re here because you googled the above error, your problem is rooted in this particular bit: file system may not support O_DIRECT. Your problem is that Cache setting on your Hard Disk entry (under VM -> Hardware) is set to “Default (No cache)”. Set it to Write Back or Write Through and it’ll launch fine.

Okay, so the VM starts with no network card - I went ahead and just added the VMware vmxnet one that Proxmox offers, since that’ll already have drivers set up on this VMware image. After that it was a few seconds to join the domain, pop the Computer object into the correct OU, run a gpupdate, and fire up my RDP connection using Remmina… and just like that, I’ve got a Windows machine set up to start playing!

Now that I’ve shaved that particular set of yaks, I’m ready to start trying to do what I wanted to do earlier this morning. Let’s see how tomorrow goes…

Using Fail2Ban to insta-block malicious hits

Tue, 07 Apr 2020 00:00:00 +1200

Okay so a few days ago I posted about adopting Caddy.

Of course, anyone who hosts anything knows that accessible services on the internet will start getting hit by scanners within literal moments of being accessible. On the very first night of having my site live on the new VPS, Logwatch informed me of a bunch of attempts to access wp-admin.php. Sigh.

Well, this entire site is built with my favourite static site generator Pelican. That means I don’t have any PHP on my site whatsoever. Why not just… block anyone that tries to open a PHP page at ericlight.com? Fail2Ban to the rescue.

Two things we need: A Fail2Ban filter, and a Fail2Ban jail.

The Filter

nano /etc/fail2ban/filter.d/caddy-php.conf (note, the name of this file must be the same as the identifier in your jail.local file)

[Definition]
failregex = ^<HOST>.*\.php(\s|\?.*)HTTP.*$
ignoreregex =

… that will catch anything in your logs such as:

nn.nn.nn.nn - - [07/Apr/2020:23:20:39 +1200] "GET /wp-admin.php HTTP/2.0" 404 0
nn.nn.nn.nn - - [06/Apr/2020:10:12:24 +1200] "GET /index.php?s=index/%5Cthink%5Capp/invokefunction&function=call_user_func_array&vars%5B0%5D=phpinfo&vars%5B1%5D%5B%5D=1 HTTP/1.1" 404 0
nn.nn.nn.nn - - [06/Apr/2020:12:51:31 +1200] "GET /wp-login.php HTTP/1.1" 404 0

The Jail

nano /etc/fail2ban/jail.local

[caddy-php]
port = http,https
logpath = /var/log/caddy/access.log
enabled = true
maxretry = 1

Note I used maxretry = 1 there. This means if an IP triggers that filter a single time, they’ll be blocked immediately… So you probably don’t want to do this just willy-nilly.

The Result

With those two parts done (and the obligatory service fail2ban reload, of course), you should find that your web server immediately bans any host that tries to load a .php file from your website!

# tail -f /var/log/caddy/access.log /var/log/fail2ban.log
==> /var/log/caddy/access.log <==
1xx.1xx.1xx.11 - - [07/Apr/2020:23:25:55 +1200] "GET /arse.php HTTP/1.1" 404 0
1xx.1xx.1xx.6 - - [07/Apr/2020:23:25:57 +1200] "GET /arse.php?1234 HTTP/1.1" 404 0
1xx.2xx.6x.1xx - - [07/Apr/2020:23:26:03 +1200] "GET /blargh.php HTTP/2.0" 404 0

==> /var/log/fail2ban.log <==
2020-04-07 23:25:55,482 fail2ban.filter [452]: INFO [caddy-php] Found 1xx.1xx.1xx.11 - 2020-04-07 23:25:55
2020-04-07 23:25:55,551 fail2ban.actions [452]: NOTICE [caddy-php] Ban 1xx.1xx.1xx.11
2020-04-07 23:25:58,309 fail2ban.filter [452]: INFO [caddy-php] Found 1xx.1xx.1xx.6 - 2020-04-07 23:25:57
2020-04-07 23:25:58,782 fail2ban.actions [452]: NOTICE [caddy-php] Ban 1xx.1xx.1xx.6
2020-04-07 23:26:03,627 fail2ban.filter [452]: INFO [caddy-php] Found 1xx.2xx.6x.1xx - 2020-04-07 23:26:03
2020-04-07 23:26:04,010 fail2ban.actions [452]: NOTICE [caddy-php] Ban 1xx.2xx.6x.1xx

Great success! You can use the banTime directive to adjust how long these blocks should last for - I’ve got mine set up to block for an hour.

Edit 2020-04-10:

Eh I realised that my RegEx missed these lovely things:

[ip redacted] - - [08/Apr/2020:16:14:10 +1200] "GET /wp-config.php.new HTTP/1.1" 404 0
[ip redacted] - - [08/Apr/2020:16:14:14 +1200] "GET /wp-config.php.old HTTP/1.1" 404 0
[ip redacted] - - [08/Apr/2020:16:14:17 +1200] "GET /wp-config.php.bak HTTP/1.1" 404 0
[ip redacted] - - [08/Apr/2020:16:14:19 +1200] "GET /wp-config.php.backup HTTP/1.1" 404 0
[ip redacted] - - [08/Apr/2020:16:14:22 +1200] "GET /wp-config.php.save HTTP/1.1" 404 0

… so yeah I just changed it to drop anything with .php in the URL. I’ll try to remember not to post any articles with .php in the URL lol.

Credits

Thanks to Phage and Xyphoid for the help in fine-tuning my rusty RegEx!

Moving to the Caddy web server

Sun, 05 Apr 2020 00:00:00 +1300

For the last couple of years I’ve been running this site, as well as my friend’s site (Under The Umbrella) on Nginx. Recently my VPS host decided to do away with their cheapest tier, so instead of doubling my annual cost, I hopped onto https://www.lowendbox.com and found myself a replacement Cheaps McGee VPS to host this.

Well, a major change like that is a great time to learn about something new, so I took the opportunity to get started with Caddy. If you don’t already know about Caddy, it’s a fast, simple, clean web server. It’s written in Go, so it’s both fast and memory safe. And hey, it’s super simple.

I’m not going to go into a whole lot of detail about setting up Caddy - there are enough tutorials out there already, and really I got all the info I needed from the website. But here are some particularly notable bits:

The Caddyfile

This lives in /etc/caddy/Caddyfile:

ericlight.com, www.ericlight.com {

 file_server
 root * /var/www/ericlight.com

 import /etc/caddy/caddy_security.conf

 log {
 output file /var/log/caddy/access.log
 format single_field common_log
 }
}

undertheumbrella.co.nz, www.undertheumbrella.co.nz {

 file_server
 root * /var/www/undertheumbrella.co.nz

 import /etc/caddy/caddy_security.conf

 log {
 output file /var/log/caddy/utu_access.log
 format single_field common_log
 }
}

And, /etc/caddy/caddy_security.conf contains:

header {
 Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
 X-Xss-Protection "1; mode=block"
 X-Content-Type-Options "nosniff"
 X-Frame-Options "DENY"
 Content-Security-Policy "upgrade-insecure-requests"
 Referrer-Policy "strict-origin-when-cross-origin"
 Cache-Control "public, max-age=15, must-revalidate"
 Feature-Policy "accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'self'; camera 'none'; encrypted-media 'none'; fullscreen 'self'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture *; speaker 'none'; sync-xhr 'none'; usb 'none'; vr 'none'"
}

** Update 2022-01-05:** Previously I’d used header / { above; that should have been simply header {. Thanks @JoHoffmann8 for pointing this out! It’s also worth mentioning that Caddy are deprecating the common_log log format, which makes me sad in one way, but I do get it - the Caddy json log file format is far richer, but I liked the easily-ingested syslog format. ¯\_(ツ)_/¯

Ok so here’s the thing. Caddy really seems to implement Python’s ethos of “Batteries Included”. The above contents are enough on their own to:

Host two separate static websites
Offer two subdomains for each of these websites
Manage the entire certificate creation and renewal process from Let’s Encrypt for two sites, plus subdomains.
Get an A+ rating on both SSL Labs and SecurityHeaders.io(!!)
And, of course, zoink all the logs into separate files under /var/log/caddy

Creating a Caddy Service file

UPDATE 2020-05-07: With the release of Caddy 2.0, it appears a regular dpkg -i caddy.deb will take care of creating the caddy.service file

If you’re running Debian, you’ll need to create yourself a service file for systemd, so you can get your server to launch Caddy on boot. I got mine from https://github.com/caddyserver/dist/tree/master/init:

/etc/systemd/system/caddy.service:

# This service file requires the following:
#
# 1) Group named caddy:
# $ groupadd --system caddy
#
# 2) User named caddy, with a writeable home folder:
# $ useradd --system \
# --gid caddy \
# --create-home \
# --home-dir /var/lib/caddy \
# --shell /usr/sbin/nologin \
# --comment "Caddy web server" \
# caddy
#
# 3) Caddyfile at /etc/caddy/Caddyfile that is
# readable by the caddy user
#

[Unit]
Description=Caddy Web Server
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --config /etc/caddy/Caddyfile --environ
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

Preventing scans

Everything above is already pretty secure - Caddy is really good at making security super easy. On top of that, Caddy is the only service hosted on this box, there’s no dynamic code (all raw HTML and CSS, thanks to Pelican), and the only things listening to the internet are SSH and Caddy itself. But even then, I get tired of seeing hundreds of scan reports every day. Fail2Ban to the rescue.

/etc/fail2ban/filter.d/caddy-4xx.conf:

[Definition]
failregex = ^<HOST>.*"(GET|POST).*" (404|444|403|400) .*$
ignoreregex =

/etc/fail2ban/jail.local:

[caddy-4xx]
port = http,https
logpath = /var/log/caddy/access.log
 /var/log/caddy/utu_access.log
enabled = true
banTime = 3600
findTime = 600
maxretry = 5

Fin!

And that’s all! I had another tweak or two to my Pelican Makefile, to point rsync to the right server, but overall that was an incredibly simple process. The Caddy team have done a spectacular job.

Lockdown Chicken

Sun, 29 Mar 2020 00:00:00 +1300

It’s the first weekend of New Zealand’s COVID-19 lockdown, and having already been working from home for a week I decided I’d get energetic in the kitchen.

Last night, with some guidance from my mate Nevyn, I put a couple chickens into a brine. I’ve brined before, but this one was super experimental for me, as usually my brines are just that one top result on AllRecipes.com, which is about as inspired as plain mayonnaise on pizza.

Well, holy shit I just finished dinner and it was good enough that it deserves a blog post. The flavour of the brine just showed through, the post-brine sauces were outstanding, and I did a pumpkin that I’ll mention below too!

The Brine

A brine is mostly there to ensure you get a super-soft and moist chicken at the end of the cook. For this purpose, only water and salt are really necessary. However you can also use the brining time to subtly introduce some flavours!

6 litres warm water
½ cup salt
⅓ cup sugar
4 star anise
2 cinnamon quills
12 allspice
10 cloves
About two tsp peppercorns
2 dried Kashmir(?) chillies

And, the méthode:

Mix your water, salt, and sugar until completely dissolved.
Put the spices into a pan on medium heat, and wiggle them around until the aroma really starts. Make sure you don’t singe them!
Add some of your brine to the pan and simmer the spices for five minutes or so, to get that fresh flavour out into the water.
Tip the whole lot back into your pot of brine, making sure you don’t leave any flavour behind. It’s ready!
Pop your chickens in, and leave them overnight. Weight them down with something if needed, and make sure you turn or mix them a couple times.

The Rub

One one of the chickens, I used Leena Spices’ exquisite Portuguese Spice Mix with heaps of crushed garlic and lemon juice. On the other chicken, I decided to make up my own. Here’s what I did:

1 tsp salt
1 tsp garlic powder
1 tsp smoked paprika
2 tsp sugar
2 tsp Worchestershire sauce
1 Tbsp oil (adjust as desired for thickness)
pinch of cayenne pepper

Just add as much oil as you need to thin the mix out enough to spread on your chicken. Also, add enough cayenne pepper for it to actually be fairly hot; it will cool during cooking. I had mine “comfortably warm”, but after cooking it was entirely unnoticeable.

The Pumpkin

This was another spur-of-the-moment thing, but it was absolutely a star of the meal. Real basically:

Cut your pumpkin in half, along the latitude, to make a couple bowls. (e.g. not from stalk to base)
Scoop out the soft inside bits of the pumpkin and discard.
Take a knife and score the inside of the pumpkin bowls with cross-cuts, as deeply as you can manage.
With your finger, just rub about 2 teaspoons of honey around every exposed part of the pumpkin flesh.
Lay some strips of butter around the rim, with an extra pat of butter in the middle.
Into the smoker it goes! I think mine was in for about two hours today.

The honey and butter will melt and flow into the cross-cuts you made in the pumpkin, allowing the flavour to penetrate.

delicious" width="1536">

The End

Here are the finishing photos! It was really a tremendous meal, and we’ve got enough left-over to make wraps for everyone tomorrow!

Eric's BBQ sauce

Sun, 22 Mar 2020 00:00:00 +1300

Quite a long time ago, I posted my recipe for smokey BBQ-baked beans. It’s still one of the only recipes I’ve seen which starts with dry beans, and includes slow-cooking in a smoker. Most of the recipes I’ve found are either for oven baking, or start with a literal can of baked beans (wtf).

Well I’m making baked beans again for the first time in ages, and I’ve taken the time to write down my tweaks. This is primarily around my BBQ sauce, hence the title of this post!

Just a few general reminders about baked beans, and then we’ll get into it:

Soak your beans the night before
Make a barbecue sauce
Simmer your beans for a bit in a normal pot
Add your aromatics and BBQ sauce after the beans are simmering
Cook your beans for a few hours until noms

The BBQ Sauce recipe

OK here’s the sauce:

350ml tomato sauce / ketchup / catsup. I used Watties, which is a fairly sweet ketchup with quite low acid
200ml brown sugar - the darker the better. You could substitute in some molasses into white sugar if you need
100ml apple cider vinegar
80ml Worchestershire sauce

That’s your base sauce. Put them all together in a pot, and bring to a slow simmer while stirring until it’s all mixed and smooth.

At this point, you want to start tasting and tweaking it to your tastes. Work out your preferred balance between the zest of the apple cider vinegar, the sweetness of the sugar, and the meaty richness of the Worchestershire sauce.

Once you’ve got your balance nailed, expand into the flavours:

1Tbsp paprika
1Tbsp English mustard
3 drops liquid smoke
1tsp of adobo sauce, if you can find it
alternatively, some cayenne or chilli powder

Be careful about your mixing method - take a quarter cup or so of your simmering sauce out, put your tablespoon of paprika into that, mix it up, and then add it to your pot. Otherwise you’ll get chunks of powder that are a pain to mix in.

Adobo sauce is pretty … interesting. It turns out, if you have a tin of chipotle peppers, they’re almost certainly bathing in adobo sauce. It’s also pretty spicy, so you really don’t use much, but it adds a whole lot of flavour that is beyond just heat. A teaspoon should be a pretty good starting point though, and you can ramp up from there as needed - it does lose a lot of it’s heat while it simmers.

Artsy Moose artisan shaving soap review

Wed, 18 Mar 2020 00:00:00 +1300

(Reposting from Facebook - originally written 2020-02-19)

Tonight I’ve got something a bit special for my #shavingPost… A few weeks ago, I offered to help beta test a new product - a shave soap in a cup - for Grere Coutie of The Artsy Moose.

The Artsy Moose is actually a really groovy little artisan handcraft shop, physically located here in Hamilton, but Grere sells most of her work online as well. Primarily focussed on soaps, there are all sorts on offer - from activated charcoal soap, to gardeners soap, to BEER soap… even soap for your doggo!

I received the cup yesterday and tonight was my opportunity to take it for a spin!

My setup tonight was my trusty RazoRock Slant, loaded with a Derby Green blade that’s a bit worn-in, having just finished it’s third shave. I used my synthetic silvertip shaving brush that’s been in my rotation for a few years now.

My regular routine for shaving is to hop in the shower, rinse my face, wash and rinse my hair, wash and rinse my face, and then bust out the shaving gear. This way my stubble has some time to soften up while I’m still being nice and efficient.

The first thing I noticed, of course, was the scent! This is a truly enjoyable scent, a bit reminiscent of sandalwood: elegant, woody, and a little musky. It could be a little bit stronger, but I like my scents quite bold, and it certainly isn’t weakly scented.

So I gave the soap block a brief rinse in warm water, drained it thoroughly, and got my shaving brush in there. After a few moments I had a beautiful white lather and got to work. I did find the lather to be very short-lived; even though I applied it fairly thickly, by the time I finished one side of my face, the other side had thinned enough that I could see skin. On my second pass I REALLY worked the brush and, although the lather came out thicker, it vanished similarly quickly.

Nevertheless, I persisted (President Warren, 2020 🙌)… The second pass was all good, but by the third pass I was feeling… Quite exposed to my blade. It just felt like I was lacking all the cushioning I’m used to from my regular creams. I didn’t let this deter me, and I continued shaving until I achieved my normal closeness of shave.

Folks I gotta say, I was really expecting to have razor burn when I got out of the shower. Persisting with a regular shave while not feeling the cushioning - and with a quickly-diminishing lather, really had me nervous. HOWEVER, I was surprised to find that nothing dire came to pass! I got out of the shower, did my dental hygiene routine, and by then I’m usually feeling a growing burn if my shave was off. But… No burn, no nicks, no cuts. I probably didn’t even need aftershave.

But, like, I USED aftershave anyway, of course. I’m not an animal. Interestingly, my aftershave did burn a little today, which is not normal. I don’t think it got along well with the shave soap, perhaps.

Overall score for the first round: Scent, five stars Cushion, two stars Lather, two stars Shave quality, four stars Face happiness… Actually a full 4.5 stars!

Next shave, I’m going to try my actual badger hair brush, as the lather may have been affected by the synthetic hair. I’ll also try a new (corked) blade and a different aftershave, to compare results. I’m also going to get reports from the rest of the family to see how it works with their shaving routines.

And that’s not all! I’m sure that Grere will be modifying the recipe in response to this review, and I think it’ll only get better as we work through!

[Update 2020-03-18]

I’ve used this shaving soap quite a few times now! I still find the same - the lather vanishes very quickly, and it still feels very thin… but it still gives actually a superb shave and I’ve never given myself any nicks or cuts while using it. I haven’t managed to convince anyone else to try it, but I’m happy it’s in my collection.

I sent this review onto Grere, and as expected, she knew exactly what to tweak:

“It sounds like the recipe could do with more stearic acid, this should provide the more stable later and cushioning that you prefer. I will whip up another batch for you to compare at some stage. However, stearic acid is derived from palm oil, which I don’t use in my soap - so i’ll see if I can try and alternative.”

So, there you go! I’ll be sure to update this post in the future, but for now, this single shave soap is going to last me at least a year. 🙂

Where does mass come from?

Wed, 18 Mar 2020 00:00:00 +1300

I’m a bit over halfway through a brilliant set of particle physics lectures by Don Lincoln, called The Theory of Everything: The Quest to Explain All Reality. Don has spent many years as an experimental particle physicist, and has worked with particle accelerators at both at Fermilab and at CERN. You may remember him from his co-discovery of the top quark in 1995, or from his membership on the CERN team when they discovered the Higgs boson in 2012. Or heck you might have seen one of his TED talks. Either way you have this set of lectures, Don Lincoln, and Don’s moustache to thank for today’s episode.

This fascinating (to me) lecture series takes us on a journey through the four known forces of the universe – the strong nuclear force, the weak nuclear force, the electromagnetic force, and the gravitational force. I’ve spent literally hundreds of hours reading on this esoterica (glares pre-emptively at autocorrect), and I know it’s a baffling area, so I’m really just going to skirt around the edges of this. But one particularly outstanding part really got my interest, and I think I can write it in an understandable form here. Wish me luck… ☘️

This post is on the topic of mass. To start looking at this we go down to the foundations of matter – let’s take a quick journey through mass:

What are things?

To get us started, I’ve put the below into a bulleted list – every level is the next step down in matter:

You, a standard human being, are made up of molecules (mostly water), along with a collection of bad ideas and faulty memories. I’m not judging.
- Those molecules in your body are made up of atoms – mostly carbon, hydrogen, and oxygen.
  Let’s look closer at hydrogen. *squints*
  - A hydrogen atom is the simplest atom. It has the atomic number of 1, meaning its nucleus has only a single proton, which is balanced by a single electron.
    
    Side-note: Hydrogen is a little special because it doesn’t have a neutron in its nucleus. In contrast, heavy hydrogen – known as deuterium – has a proton, an electron, and a neutron.
    - A proton is, in turn, made up of three quarks, and weighs roughly 938 MeV/c². (that’s megaelectronvolts… more below)
      - An up quark (carrying a positive electric charge of “plus two-thirds”) has a mass of roughly 2.3 MeV/c² (ignore that part)
      - The down quark (carrying a negative electric charge of “minus one-third”) has a mass of roughly 4.8 MeV/c² (ignore that too, the fun bit is coming)
        
        (Quarks appear to be made up of strings, but that’s too deep for this article. Extra credit: String theory, superstring theory, and M theory)
        
        Holding the quarks together are the Gluons, which are the particles which mediate the strong nuclear force. These are massless*-ish*, coming in at almost exactly 0 MeV/c².
      - A proton is made of two up quarks and a down quark, which gives it a total electric charge of +1. (four-thirds from the up quarks, minus a third from down quark)
    - The electron is not made up of quarks – it appears to be a primary particle (specifically, a lepton) that cannot be subdivided. It has an electric charge of -1, and a mass of about 0.5 MeV/c².

What’s a MeV, and has it been approved by the FDA?

Okay, so you’ve probably never seen mass described in MeV/c² before. That’s because people live in the world of Relatively Big Things™, so we generally think of mass as what you might measure on a set of bathroom scales (also known as m’ass). But when you’re looking at particle physics, things get way fuzzier… SO fuzzy, in fact, that we don’t weigh things in grams – we weigh them in energy! Specifically, in electron-volts.

For example, a proton weighs 938 million electron-volts (/c²), and an electron weighs about 500,000 electron-volts. CBut… how does *that* work? How on earth do we measure mass in energy? You wouldn’t say “this glass weighs 500 calories”, would you?

HECK YEAH, you would! It’s the formula most deeply burned into your brain from high school: E=mc². Energy (in joules) = Mass (in kilograms) * the speed of light (in metres per second) squared. Einstein’s been saying it for over a hundred years… mass is energy, and energy is mass! And mass contains a whole honkin’ bunch of energy. So much energy that a single gram of matter – about the mass of a business card – looks something like this:

Energy (joules) = 1 gram * speed of light^2
Energy (joules) = 0.001 * 300000000 * 300000000
Energy of 1 gram of matter = 9,000,000,000,000,000 joules

… or about 2,151,051,625,239 calories.

So yep, you can definitely weigh things in energy … and because subatomic particles are so small, energy is a totally sensible measure to use. So let’s get back to the world of the extra-tiny.

Those numbers look fishy…

Yes, those numbers are weird. And I’m not talking about my equations above, I’m talking about the mass of an atom. Check it out again:

Proton = 2x up quarks + 1x down quark = 938 MeV/c²

But the two up quarks only bring 4.6 MeV/c² to the party, and that down quark is only 4.8 MeV/c², adding up to a mere 9.4 MeV/c². In fact, when you put together the mass of all the components of a proton, you still only see about 2% of the mass of the proton. So… where’s all that other mass coming from?

Well, okay. Those quarks aren’t just sitting stationary – they’re moving around at nearly the speed of light, within an incredibly tiny area. In fact, these quarks are fizzing around in an area approximately one-quadrillionth of a meter.

And here’s the whole idea that got me writing this post:

Our quarks, zooming around near the speed of light, and confined within such an incredibly tiny area… Well heck, that’s a whole lot of energy.

And what is energy?

It’s mass.

And that is where most of the mass of an atom comes from. From this kinetic energy, found within in the quantum foam, and expressing itself in our universe as mass.

fin.

Fail2Ban+Nginx (blocking repeated 404's, etc)

Mon, 16 Mar 2020 00:00:00 +1300

After hosting a mail server for a few years, I’ve gotten tired of seeing alllll the 404 attempts in my daily logwatch. Fail2Ban can help here really well, and it turns out to be really easy.

Start with nano /etc/fail2ban/filter.d/nginx-4xx.conf (I’m counting on your running Debian and having things in default locations here), and enter the following:

[Definition]
failregex = ^<HOST>.*"(GET|POST).*" (404|444|403|400) .*$
ignoreregex =

Note: Make sure you use a capital ‘D’ in Definition there.

Now, edit your /etc/fail2ban/jail.conf, and add these lines:

[nginx-4xx]
enabled = true
port = http,https
logpath = /var/log/nginx/access.log
maxretry = 3

Of course, modify your maxretry as desired. Refresh your Fail2Ban rules with service fail2ban restart, and it should now be on the lookout for repeated 4xx errors.

You can confirm it’s working with: fail2ban-client status nginx-4xx:

Status for the jail: nginx-4xx
|- Filter
| |- Currently failed:	2
| |- Total failed:	9
| `- File list:	/var/log/nginx/access.log
`- Actions
 |- Currently banned:	2
 |- Total banned:	2
 `- Banned IP list: 123.232.123.101 123.232.123.102

Success!

Worth mentioning… for the first while I thought it wasn’t working, but I realised I was trying to trigger the rule from on the same LAN, and …

[DEFAULT]
# time is in seconds. 3600 = 1 hour, 86400 = 24 hours (1 day)
findtime = 3600
bantime = 3600
maxretry = 5
ignoreip = 127.0.0.1 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16

… I was hitting it from within the ignoreip range. 😬

And of course, I can’t finish off a blog without giving credit where it’s due. I got this all from GitHub user AysadKozanoglu, here: https://gist.github.com/AysadKozanoglu/1335735272fb3b00a03bd3eea22af818

sa-compile failing during upgrade

Sun, 15 Mar 2020 00:00:00 +1300

I just spent an hour troubleshooting the most ridiculous thing. I guess to help people search, I’d describe this as “dpkg failing at sa-compile in Debian 10” but that’s really not a good picture of what’s happening here…

Setting up sa-compile (3.4.2-1~deb9u3) ...
Running sa-compile (may take a long time)
/bin/sh: 1: x86_64-linux-gnu-gcc: Permission denied
make: *** [body_0.o] Error 126
command 'make PREFIX=/tmp/.spamassassin22062Ifq5yDtmp/ignored INSTALLSITEARCH=/var/lib/spamassassin/compiled/5.024/3.004002 >>/tmp/.spamassassin22062Ifq5yDtmp/log' failed: exit 2
dpkg: error processing package sa-compile (--configure):
subprocess installed post-installation script returned error exit status 25

<-- snip -->

Errors were encountered while processing:
 sa-compile
E: Sub-process /usr/bin/dpkg returned an error code (1)

In my defence, it seemed similar to a recurring issue I’ve had on this server in the past, (which I now promise to address in a separate post). That’s why I ended up on a red herring hunt.

Anyway, I spent a bunch of time trying to troubleshoot dpkg and Python, before realising that a few months ago I’d installed Lynis and worked through a bunch of the hardening recommendations.

… One of which is restricting the execute permissions on /usr/bin/gcc. Which is a symlink to /usr/bin/gcc-6. Which is a symlink to x86_64-linux-gnu-gcc-6.

root@x:/usr/bin# ls -l x86_64-linux-gnu-gcc-6
-rw-r--r-- 1 root root 949016 Feb 15 2018 x86_64-linux-gnu-gcc-6

Yes. I was experiencing problems compiling things because I’d removed execute permissions for all users on the compiler I needed, and I’d spent an hour ignoring the “Permission denied” error that told me where to look.

Fixed with a simple chmod 744 /usr/bin/x86_64-linux-gnu-gcc-6.

FreeNAS finally supports ECC crypto!

Sun, 02 Feb 2020 00:00:00 +1300

Woo! FreeNAS finally supports ECC crypto for it’s SSL certificates in 11.3!

I reported this as a bug almost exactly two years ago, and kept having to do nasty workarounds to make it work every time I installed an update. Very happy that it’s finally sorted and I can use my proper shiny elliptic curve SSL setup! :)

Fortinet SSL VPN Certificate extensions

Sun, 19 Jan 2020 00:00:00 +1300

If you’re setting up a new SSL VPN with certificate authentication, and if you already have an internal PKI, you’re probably considering using your internal certification authority to create the SSL keys and certificates for your VPN clients. Awesome! This is a great and easy way to do this job.

But there’s a dearth of information on the Interwebs regarding what Extended Key Usages you need to have enabled in your Certificate Template for these. There are a heap available, including Email Signing, Timestamping, Code Signing, IPSec SSL things (about 8 of these), and way more options.

Well, your search can end now. There’s only one Extended Key Usage option required for SSL certificate authentication with the Fortinet FortiClient VPN:

Client Authentication

That’s it. That’s the post.

What's so weird about lightspeed, anyway?

Thu, 16 Jan 2020 00:00:00 +1300

Have you ever experienced that thing where you’re in your car, stopped at a traffic light - you suddenly get the sensation that you’re rolling backward, so you slam on your brake… but you already had your brake on, and you weren’t moving! It was the car beside you rolling forward, but your brain misinterpreted it as your own - backwards - motion.

So, another mental image: imagine that you find yourself suddenly transported to a quiet, peaceful mountaintop. There’s a monastery here, and Monks are quietly meditating. There is no sound, no wind, just silence and calm. You would probably say you’re standing perfectly still.

Now, remembering that perception of stillness, think about the example in this tweet:

Take a kid & explain to him the true fact that the Sun is so far away that it would take a car going 100 mph a 100 years non-stop to get there. And then tell him to picture that same distance BEYOND the Sun! And then tell him that he and you were at that very spot 6 months ago.

– @KevinPaulGregg

… your peaceful mountaintop really wasn’t that still at all. In fact, it was hurtling through the galaxy at … (cough) … an astronomical rate. And on top of that, it’s spinning wildly: at the equator, the earth’s surface is moving at almost 1700km/h.

This speaks to the very basis of an important question: What is speed?

Over 400 years ago, Galileo Galilei’s Principle of Relativity (not to be confused with Einstein’s Theory of Relativity) emphasised that the concept of speed requires a frame of reference. That is, speed is always relative to something else. If you’re driving at 100km/h, that speed is relative to the road under your tyres - but compared to a motorbike passing you at 120km/hr, you’re moving backwards. Or … from a different perspective, you’re sitting still, and the motorbike is only travelling at 20km/hr. And no matter how fast you’re driving, your steering wheel isn’t really moving relative to you.

OK I think I’ve explained that point to death now. Speed is always measured between two points in space. But here’s the interesting bit.

Maxwell’s Equations, published in the 1860’s, can be used to calculate the speed of light - exactly 299,793,458 metres per second in a vacuum. But there’s something missing: that all-important frame of reference; the OTHER point in space to measure the speed against. What’s going on there? Lots of theories and experiments were floated to figure out where this missing factor was, but everything was shown to be incorrect… that is, until Einstein looked at the problem.

Einstein - in typical Einstein fashion - simply looked and said, “if the speed of light doesn’t have a frame of reference, that must mean that the speed of light is the same in any frame of reference”. That is, if you shine a torch forward, the light is travelling away from you at 300,000km/s. If you keep the torch on and hop into a speeding car, the light is still travelling away from you at 300,000km/s. If you turn around and point the torch out the rear window, the light is travelling away from you at 300,000km/s… and towards someone else, still at 300,000km/s. So what’s happened to the speed of your car??

Okay, so the speed of light is weird. But the weirdest part is what happens when you look at what that means. Let’s go back to my earlier question – what is speed? In one way, this is easy: speed is simply the distance travelled over a certain duration. But take a closer look at what those words mean: distance refers to space; duration refers to time. So, speed is the rate of movement through spacetime. Fine, semantics, whatever.

But these semantics matter. Because what we’re seeing here isn’t just light being weird: now we’re actually seeing space and time moving weirdly - dancing around themselves in a big conspiracy to force the speed of light to be 300,000km/s, no matter what your frame of reference is.

It’s time to wrap up this episode - mostly because I’ve reached the limits of my knowledge. Earlier in this article, I asked “what’s happened to the speed of your car” and, frankly, I have no idea. When it comes to the warping of spacetime I get really confused. If you’re travelling in a train at 200,000km/s, and point a powerful laser in the direction of travel, that laser’s light is going 300,000km/s relative to you. But … then we get silly business, like Lorentz contraction squishing the length in the direction of travel, time dilation starts messing with things, and we get space bending. So to the outside observer, the laser is going … 300,000km/s. ¯\_(ツ)_/¯

“People assume that time is a strict progression of cause to effect, but actually — from a non-linear, non-subjective viewpoint — it’s more like a big ball of wibbly-wobbly… timey-wimey… stuff.”

– The Tenth Doctor, Doctor Who, “Blink”

The tension between classical and relativistic mechanics

Thu, 09 Jan 2020 00:00:00 +1300

When I was considering writing my post on time dilation, I expected precisely zero feedback… just on the basis that nobody would read it. I’m really pleased with the feedback I actually got, and you’ve all spurred me to write another post on the topic. So today, I’m going to give you a brief glimpse into what caused all this ruckus: Einstein’s general theory of relativity.

If you remember, Einstein first came up with the special theory of relativity in 1905, and it took another ten years to get to the general theory of relativity. This alone should show you that the jump from the special theory to the general theory was a major leap - but let’s step back and see what this is all about.

Our very good childhood friend, Isaac Newton, published his Prinicpia in SIXTEEN EIGHTY SEVEN, which is… like… aaaaages ago. With this, he laid the foundation of “classical mechanics”, which are the physical concepts and mathematical methods that describe the motion of objects under a variety of forces.

Classical mechanics worked well - really well. There were tweaks here and there there over the ~~centuries~~, but overall, classical (or “Newtonian”) physics had things pretty much nailed… That is, until we started thinking about things that were really heavy, really small, or really fast… at that point, classical mechanics started to fray at the edges.

One of the earliest recognised failures of classical mechanics was the “precession of the perihelion of Mercury”. Don’t worry what those words mean; basically, Mercury’s orbit didn’t behave quite the way that classical mechanics said it should. The effect is suuuuper tiny: classical mechanics says that Mercury’s orbit should change by 5557 arcseconds per century, but in reality it changes by 5600 arcseconds per century… a difference of 43 arcseconds.

How big is 43 arcseconds? It’s about two-thirds of one degree. If you stand on a flat field and imagine a soccer ball on the ground, a kilometer away, the angle drawn from the ground under your feet to the top of that soccer ball is bigger than the 43 arcsecond discrepancy.

… I’m gonna say that again in case you missed it: that was per century. Less than a kilometre-away-soccer-ball every hundred years. Yikes.

Where did this discrepancy come from? It all started from a very fair assumpiton of Isaac Newton’s: he perched his calculations on the basis that things were consistent. Well, the universe Does Not Roll That Way™. Space warps; time stretches; light falls into gravity wells. But because these effects aren’t really visible in our day-to-day life, it took almost two hundred years - and the genius of Einstein - before classical mechanics was overturned by the general theory of relativity in 1915.

Okay this has become a little long and rambling now, so I’m going to take a break and come back to the topic next week! :)

Zentyal 6.0 to 6.1 upgrade getting stuck

Tue, 17 Dec 2019 00:00:00 +1300

So I faced a little challenge with a Zentyal server the other day. I was upgrading from ye olde 6.0 to 6.1, when everything just stopped. I let it sit in the corner for about an hour or so, but it never picked up the thread. All the services were still live, so I logged in to have a look.

(Note, my DNS server is named RIMU; yours may be something else!)

Running ps aux, I discovered this line:

:::bash
sh -c /usr/bin/sudo -p sudo: /var/lib/zentyal/tmp/x2M7gkZVvm.cmd 2> /var/lib/zentyal/tmp/stderr

So, I had a quick look at the contents of that stderr file:

:::bash
# cat /var/lib/zentyal/tmp/stderr 
Password has expired
dns-RIMU@ad.ericlight.com's Password:

And what does that temp .cmd file contain?

:::bash
# cat /var/lib/zentyal/tmp/x2M7gkZVvm.cmd 
kinit -k -t /var/lib/samba/private/dns.keytab dns-RIMU

Running that kinit command indeed prompts for a password reset, but the interesting thing is that samba-tool shows me this password shouldn’t expire:

:::bash
# pdbedit -u dns-RIMU -v | grep change
Password can change: Wed, 31 Oct 2018 21:47:30 NZDT
Password must change: never

… riiiiiiight, that’s a bit interesting. And yet I’m still being prompted to set a new password. I used samba-tool to remind samba that this password shouldn’t expire:

:::bash
# samba-tool user setexpiry dns-RIMU --noexpiry
Expiry for user 'dns-RIMU' disabled.

And now…

:::bash
# pdbedit -u dns-RIMU -v | grep -i change
Password can change: Wed, 31 Oct 2018 21:47:30 NZDT
Password must change: Tue, 19 Jan 2038 16:14:07 NZDT

Now I can run kinit against dns-RIMU perfectly fine, and indeed the Zentyal upgrade succeeded!

StarCraft Remastered broken under Wine

Mon, 09 Dec 2019 00:00:00 +1300

I know this post is only going to be relevant for a couple of weeks, but today I tried to launch StarCraft Remastered under WINE and it crashed.

Pertinent info: StarCraft Version 1.23.2.6926 WINE version 4.20-staging Battle.Net version 1.18.0.11698

Fix was to open the Battle.Net settings -> Game Settings -> “Launch 32-bit client (instead of 64-bit)”

Apparently the 64-bit version has become the default sometime since I last played it (October?!): https://us.forums.blizzard.com/en/starcraft/t/enabling-64-bit/458/7

Spacetime, and Time Dilation

Sun, 10 Nov 2019 00:00:00 +1300

I remember reading Stephen Hawking’s A Brief History of Time back when I was about 16 years old - a bit over 20 years ago. I didn’t even come close to understanding it then. I re-read it on Audible a year ago and I can report that I still didn’t understand it at all. Either way, I’ve been interested in physics and spacetime for a huge chunk of my life, and I wish I’d spent more time on mathematics when I was younger, because I’d have really loved to be a theoretical physicist. I shit you not. The universe just seems to be more predictable than computers.

Einstein published the special theory of relativity in 1905, and it took him another ten years to bring this into the general theory of relativity (which incorporated gravity) in 1915. The special theory is only “special” in the sense that it assumes a flat spacetime; it was the general theory which extended relativity to allow for the curvature of spacetime - for example, the curvature caused by massive objects.

Today I’m going to share my understanding of time dilation, based on what I’ve learned from Dr Brian Greene, Dr Michio Kaku, and my favourite astrophysicist Marcus Chown. I’ll start with a disclaimer: I don’t understand any of the calculations behind any of this. I’m familiar with some of the words, and I have a vague grasp of some of the concepts, but I’m mostly limited to regurgitating things I’ve read and then swearing loudly to distract people from asking questions.

I’m also quite conscious that this is around step three in the age-old progression:

Primary school: “This is how the weather works.”
High school: “That wasn’t quite right - this is how the weather works.”
Undergrad: “That wasn’t quite right - this is how the weather works.”
Postgrad: “That wasn’t quite right - this is how the weather works.”
Doctorate: “Nobody knows how the weather works.”

… Nevertheless, the actual concept is pretty simply stated, once we get everything laid out. In brief, time dilation is when time appears to slow down for an object, when that object is moving very quickly. There are three important parts to that:

the slowing of time;
the fact that it looks to you as if time has slowed for the object;
the object travelling quickly relative to you.

That means, if your twin travels (around Europe, say) at the speed of light for “ten years” (your time), you will age ten years, but they will hardly age at all.

That’s pretty weird - why does that happen? We can understand where time dilation comes from by remembering a few things:

We all know that the universe’s speed limit is the speed of light, c.
We’re all familiar with the three dimensions of motion.
We’ve probably heard someone say “time is the fourth dimension” (aka “Minkowski spacetime”).

Here’s the thing about speed limits: they apply whether you’re moving in a straight line, or if you’re moving on an angle. Imagine a box, a hundred kilometers square. If you travel a hundred kilometers from left-to-right, you will start at one edge of the box, and finish at the other. But if you start at the top-left corner of the box, and travel towards the bottom-right corner of the box, and still travel a hundred kilometers, you will end up only about 70% of the way to your destination… because some of your motion has been ‘used up’ travelling in the up/down direction, instead of just the left/right direction.

Now, here’s the thing: you’re always travelling at c. It’s just that mostly you’re travelling through time at the speed of light. Any movement in the other three dimensions will ‘use up’ your speed through the time dimension (because your total speed can’t be faster than c). Quick examples:

You, sitting here now reading this:

Speed in x dimension: 0km per second
Speed in t dimension: 299,793km per second

You, 20 years from now, travelling away from Earth at a currently-safe velocity of 3G’s (29.42 meters per second):

Speed in x dimension: 0.02942km per second
Speed in t dimension: 299,792.97058km per second

You, 200 years from now, travelling away from Aldeberan at a somehow-safe-by-then velocity of 300G (2942 meters per second):

Speed in x dimension: 29.42km per second
Speed in t dimension: 299,763.58km per second

You, never, travelling away from the milky way at 95% of the speed of light:

Speed in x dimension: 284,803km per second
Speed in t dimension: 14,989km per second

So, as you get faster in one dimension, you necessarily must move more slowly through time! Time dilation comes down to that, plus the hard-to-grasp concept that you’re only either accelerating or stationary in your own frame of reference… but that’s a story for another day.

That pretty much exhausts everything I know about time dilation… I hope you enjoyed reading it. Huge thanks to Ellie for finding me a friendly physicist to check over this for me!

In case you’re looking for some good books on the science of the very fast, the very big, the very heavy, and the very small… Here are some books that I really enjoyed reading:

Brian Greene - The Elegant Universe
Marcus Chown - Quantum Theory Cannot Hurt You
Michio Kaku - Einstein’s Cosmos

Inside the Check Point 1180

Sun, 28 Apr 2019 00:00:00 +1200

I’ve got a spare Check Point 1180 here, and I’m fiddling around with it a bit. Learnings so far:

Platform: ARM
CPU / RAM: ARM926EJ-S, 512MB RAM
System board: Marvell RD-88f6281A LE

When a docking station wrecks your breakfast

Sat, 23 Feb 2019 00:00:00 +1300

It’s been a while since I posted, but we bumped into something completely batshit crazy at work that I just had to share.

One of my colleagues was setting up a new computer, and was unable to RDP into the new build. He kept getting the error message “Your session ended because there was a data encryption error. If this keeps happening, ask your admin or tech support for help.”

We’d never experienced this error before, and - although the internet knew about it - there were not very many helpful hits on Google. In fact, all of the reports we found were the result of a bug in Windows 7, and we were only using Windows 10.

In the end, completely weirdly, my colleague tried a different docking station, and it worked! This is absolutely the weirdest thing I’ve ever seen. Both the working and failing docking station had the most recent firmware applied, and they were both flashed from the same image, so it’s not a corrupted download or anything.

Metadata about this issue: Both machines were Windows 10, 1709 build, with all updates installed. The originator machine was an HP Z240 workstation, and the target machine was a Dell 7390, connected to a Dell WD15 docking station. All three items had received fresh firmware updates and all Windows Updates.

American Crew's Moisturising Shave Cream

Sat, 29 Dec 2018 00:00:00 +1300

Yanno how I said it’s hard to find American Crew aftershave? Well my better half went out yesterday and found a place in lil’old Te Awamutu that sells their shaving range: Team Seven, beside Viands. She bought me a bottle of my favourite “Post-Shave Cooling Lotion” (oh my god that name), but also got me a tube of their “Moisturising Shave Cream” to try. For shiggles.

Clearly, American Crew are better at chemical formulation than they are at coming up with names. But I digress.

Whelp I whipped out my trusty DE89, popped in a fresh Derby Extra blade, soaked my shaving brush and gave it a go.

Firstly - no. This is absolutely not made for a shaving brush. It’s hand-application all the way. It’s also not really a foam like what I’m used to - it’s very much a cream style, and it’s quite stubborn about remaining a cream, thankyouverymuch. In contrast, my Proraso or TOBS creams froth up quite heartily when I whip them with a brush, but this really resisted that.

Okay so, without making it frothy, it definitely takes more product to shave. I’m going to say double, but it might even be more. At about $32 a tube, that is definitely verging out of my price range. 2/10 on price for this.

The feeling, however, was really divine. There’s not a whole lot of scent, but there’s significant menthol action going on here, and it really gave me a refreshing zing that dwindled on my face through the shave and rinse. Delight rating 7/10.

Efficacy - this is hard for me to judge, since I was using a brush and thus really didn’t have enough on there for my second pass. So I kinda mucked up my shave, but it was much more my fault than the cream’s. Aside from that it left a beautiful coating of light avocado oil and aloe on my skin, meaning even though I under-catered on the product, I didn’t entirely munge up my face. For this, I’m giving it a decent 7/10 on efficacy. (See update below, though)

These ratings seem kinda low, but honestly this is going to be my favourite shave cream through the summer, with that nice refreshing menthol coolness. However with the price and sheer quantity needed for a good shave, I expect I’ll gravitate back to my Proraso creams once this is gone.

Update 2018-12-31: I’ve had another shave today without the brush, and just applying the cream directly to my skin. Definitely a better result, but once again I under-catered on the second pass and got myself some irritation happening. It’s certainly going to be at least double what I previously used. However once again my face feels actually quite good, and geez the menthol is nice.

Since I’ve under-catered twice now, and both times I’ve actually still had a decent shave, I think I need to upgrade this to 8/10 efficacy.

Proraso Azur Lime aftershave balm

Fri, 28 Dec 2018 00:00:00 +1300

Gather around, children, and I’ll regale you with a tale of tonight’s shaving.

My go-to razor setup at the moment is my Edwin Jagger DE89, with Derby Extra blades. But that’s not new.

You might remember that my favourite aftershave is American Crew’s “Post Shave Cooling Lotion”. I’ve been using this almost exclusively for about 5 years now, and I love it. It’s wonderfully soothing, and it’s a fairly thin balm so it’s easy to avoid overly-thick application. It’s also not as thin as a milk (such as “The Standard”, by The Gentlemen’s Refinery), which has a tendency to run through your fingers on the way to your face, not entirely unlike… milk.

Unfortunately American Crew’s aftershave can be really difficult to find. So I popped a question to the team at FineShave, and Frank gave me some insight into what he uses. This spurred me to try the Proraso Aftershave line, and today I took delivery of a bottle of Azur Lime.

This is a really, really nice aftershave. Again, it’s a balm (I don’t have any splash-type aftershaves yet). It’s a little thicker than the American Crew that I’m used to, but still not thick.

WOW does it cover a lot of distance. I used three, maybe four drops worth and it covered my whole face. This 100ml bottle is going to last forever, but it’s still a bit pricey at $35, which makes me give it a 6/10 on the price scale.

It’s a light citrusy scent, with a little bit of mint, and a slightly musky underbelly. It smells nice, but the scent wears off fairly quickly, so probably a 7/10 for scent.

I also happened to give myself a spectacularly awful shave, which this aftershave has completely addressed - no discomfort or ‘hot areas’, and only a little redness in the mirror. I think that’s a 10/10 for effectiveness.

All in all, this is a serious contender for my new favourite, and I expect I’ll buy another bottle when it’s done! Recommended.

New things I didn't know about WireGuard

Thu, 27 Dec 2018 00:00:00 +1300

This is part of my brief series on WireGuard. I’m pretty enamoured with WireGuard and the way it works, and I’ve been using it pretty seamlessly for over a year now. I’ve learned a couple things that weren’t immediately obvious though, so I’m documenting them here.

Easy Provisioning

Samuel Holland mentioned an interesting trinket, in his post at https://lists.zx2c4.com/pipermail/WireGuard/2018-December/003703.html:

"[…] WireGuard will ignore a peer whose public key matches the interface’s private key. So you can distribute a single list of peers everywhere."

You can combine this with wg addconf like this:

Each peer has its own /etc/WireGuard/wg0.conf file, which only contains it’s [Interface] section
Each peer also has a shared /etc/WireGuard/peers.conf file, which contains all the peers
The wg0.conf file also has a PostUp hook, calling wg addconf /etc/WireGuard/peers.conf

It’s up to you to decide how you want to share the peers.conf, be it via a proper orchestration platform, something much more pedestrian like Dropbox, or something kinda wild like Ceph. I dunno, but it’s pretty great that you can just wildly fling a peer section around, without worrying whether it’s the same as the interface.

Setting Private Key from a file

Another piece of learning, courtesy of Samuel Holland, at https://lists.zx2c4.com/pipermail/WireGuard/2018-December/003702.html.

You can read in a file as the Private Key by doing something like:

PostUp = wg set %i private-key /etc/WireGuard/wg0.key

Tomato Soup

Mon, 07 May 2018 00:00:00 +1200

I came home tonight to discover my wife had made probably the most amazing tomato soup I’ve ever tasted. She performed this feat with about ten minutes of effort (plus about an hour of simmering), so I decided I needed to note it down.

Ingredients

1 Tbsp oil
1 medium onion, sliced
1/2 head of garlic, peeled and chopped
About 4 slices of bacon, diced
3 cans of chopped tomatoes
2 tsp of vegetable stock powder
2 tsp of sugar
1 cup of water
Cream
Parsley for garnish

Method

Get yourself a piping-hot frying pan
Add the oil and the onion; fry until transluscent
Add the garlic and the bacon; fry until bacon starts to brown
Add chopped tomatoes, veggie stock, and water; stir
Simmer for at least 45 minutes; preferrably longer
Serve with a dollop of cream, and garnish with parsley

That’s it! It’s hands-down delicious.

When AppArmor eats your breakfast

Mon, 19 Mar 2018 00:00:00 +1300

I’ve knocked heads with AppArmor a few times now. To be quite honest with myself, I think it’s primarily because I install it, enable all the plugins, and then forget it’s there until there’s a problem.

TL;DR:

On a fully-updated Zentyal 5.0 system running DHCP, the AppArmor profile in /etc/apparmor.d/usr.sbin.dhcpd will prevent isc-dhcp-server from restarting itself after an upgrade.

More search-type words: Zentyal dhcp server doesn’t start again after upgrade. isc-dhcp-server graceful shutdown, but no restart.

This frustration-laden, Google-friendly equivalent of speaking slowly and clearly should illustrate how impossible it was for me to find guidance on this.

To fix:

# aa-complain /etc/apparmor.d/usr.sbin.dhcpd

Okay, now that the TL;DR is finished, here’s the rest of the story:

Unfortunately there’s always a problem with AppArmor eventually.

Even worse, the problem is often silent. I’ll just find that, for example, NONE of my man pages work. Or logrotate will fail. For example, here’s what happens when you enable aa-enforce mode on your friendly resident user manual:

 $ man
 What manual page do you want?
 $ sudo aa-enforce /etc/apparmor.d/usr.bin.man
 Setting /etc/apparmor.d/usr.bin.man to enforce mode.
 $ man
 $man: can't open the manpath configuration file /etc/manpath.config

If I hadn’t enabled aa-enforce immediately before that, what would you have done? You would have gone ahead, made sure that manpath.config existed, made sure it wasn’t corrupted, maybe tried running man as root… maybe even copied a manpath.config file from a known-working computer. But, hidden within dmesg is our old friend:

[149929.763064] type=1400 audit(1521449052.257:111): apparmor="DENIED" operation="open" profile="/usr/bin/man" name="/etc/manpath.config" pid=22633 comm="man" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

And that example is actually better than it used to be - quite a few times in the past four years, I’ve received this:

$ man
$

… from a fully-updated Debian system.

This makes me sad.

Or, in one particular case, ISC-DHCP-SERVER of all things will launch perfectly well, but will fail to restart after a package update, and the first thing I’ll notice is “o hai all the things is broken halp plz”. Something like that.

When you’re troubleshooting DHCP, you’ll probably check systemctl, leases table, maybe /var/log … But when you get around to checking dmesg, you may notice the root of the problem:

From dmesg -T:

[Sat Mar 17 03:44:28 2016] audit_printk_skb: 18 callbacks suppressed

Considering the update occurred at 03:44am, this is probably our problem. I can’t see for sure, but I’m pretty certain those 18 suppressed callbacks are filled with “DENIED” log lines.

The fix is simple:

# aa-complain /etc/apparmor.d/usr.sbin.dhcpd 
Setting /etc/apparmor.d/usr.sbin.dhcpd to complain mode.

And your isc-dhcp-server should now restart gracefully after it’s scheduled upgrades!

Installing PowerChute Network Shutdown on ProxmoxVE

Tue, 27 Feb 2018 00:00:00 +1300

This is going to be one of those posts where I just dump stuff. Power failures are shite, and if your server isn’t talking with your UPS, it won’t shut down gracefully. This is the story of my trying to get APC’s PowerChute Network Shutdown working on a ProxmoxVE 5.1 environment.

Note I opted out of using either nut or apcupsd, because I had a nice fancy Network Management Card (NMC2 / Schneider AP9631) available. Also, I wanted to shut down more than a single server, so a USB or Serial cable wouldn’t do the trick.

Install Java Runtime Environment: apt install openjdk-8-jre
Download PCNS onto your Proxmox server - http://www.apc.com/shop/us/en/categories/power/uninterruptible-power-supply-ups-/ups-management/powerchute-network-shutdown/N-auzzn7
Extract the download: tar -xvf pcns420Linux-x86-64.tar.gz
cd ./Linux_x64
sudo su
./install.sh
When prompted for your Java location, use /usr/lib/jvm/java-8-openjdk-amd64/jre/bin
I didn’t enable SMTP

Results in:

 root@ ~/Linux_x64# ./install.sh 
 ------------------------------------------------------------------
 PowerChute Network Shutdown 4.2.0 for Linux
 Copyright (c) 1999-2016 Schneider Electric.
 All Rights Reserved.
 ------------------------------------------------------------------

 OS=Linux

 Initializing ...

 Press any key to display End User License Agreement
 <--snip-->
 Do you agree to the above license terms? [yes or no]
 yes

 Please enter the installation directory or press enter to install to the default directory (/opt/APC/PowerChute):


 Are you sure you want to install PCNS to /opt/APC/PowerChute [Yes|No]? 
 yes
 PCNS will be installed to /opt/APC/PowerChute

 Please enter java directory if you want to use your system java (example:/usr/local/bin/jre/jre1.8.0_91) or press enter to install the bundled Java:
 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin 

 Checking version of Java ...
 Detected Java Version: 1.8.0.151
 Acceptable version

 openjdk version "1.8.0_151"
 OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-1~deb9u1-b12)
 OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

 JAVA_DIR=/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/

 Copying the installation files ...
 Extracting PCNS files ...
 PCNS is extracted to /opt/APC/PowerChute
 Configuring startup files ...
 Startup script=/etc/rc.d/init.d/PowerChute
 Updating Linux symbolic link ...
 Configure Firewall
 Configuring uninstall script ...
 Setup the m11.cfg file

 Enable SNMP Support [Yes|No]? 
 no
 SNMP Not Enabled

 PowerChute Network Shutdown, v4.2.0
 Copyright (c) 1999-2016, Schneider Electric. All Rights Reserved.
 Startup completed.


 Installation has completed.
 PowerChute Network Shutdown can be accessed through your browser at https://<your_server_ip_address>:6547
 Please complete the configuration wizard so that PowerChute Network Shutdown can protect your server.

Once you’re finished with that, navigate to the new web portal on your server (second-to-last line in the output above), and complete the setup of your PCNS instance!

When you’re configuring PCNS itself, you’ll be prompted for the User Name, Password, and Authentication Phrase for the UPS. These are the user details you use to log into your NMC.

By default, the Authentication Phrase is admin user phrase, but you should totally change this. Assuming you’ve updated the firmware on your NMC, you can set the Authentication Phrase at Configuration -> Shutdown.

Setting up your outlet groups and shutdown times are left as an exercise to the reader. :)

Getting rid of the # format in Flex

Fri, 12 Jan 2018 00:00:00 +1300

Back when I was getting this blog set up, I had a short whinge about the default way that the Flex theme created links to pages.

Specficially, creating a link to “Tuna Patties” (for example), Flex would append the link with an identical stub, such as https://www.ericlight.com/tuna-patties.html#tuna-patties

I thought this was silly, so I found a way to fix it, but never bothered submitting a Pull Request to Alexandre because I figured it was intentional.

Well, it turns out I wasn’t the only one. The good Dmytro Litvinov thought the same, and he created a Pull Request to fix it, almost a whole year ago!

Now, you can avoid this behaviour by simply adding DISABLE_URL_HASH = True to your pelicanconf.py!

Fix Dell Keyboard Backlight under Debian

Thu, 11 Jan 2018 00:00:00 +1300

My personal laptop is an old Dell Latitude E6410. One of the things I actually love about it, is the fact that the keyboard has a backlight.

However, because Linux, sometimes that backlight just stops working. When this happens, you’ll see error messages in dmesg, such as:

dell_wmi: Unknown key with type 0x0011 and code 0x01e2 pressed

If this happens to you, just run this command:

$ sudo echo 7 > /sys/devices/platform/dell-laptop/leds/dell\:\:kbd_backlight/brightness

You can echo a higher number if you want your keyboard brighter, but I found 7 to be more than bright enough.

Smokey BBQ-Baked Beans

Wed, 10 Jan 2018 00:00:00 +1300

We recently got a Fornetto Basso - it’s an 18" bullet smoker/barbecue, advertised with three tiers of deliciousness: two bbq racks, and a rail under the lid for hanging meats for smoking. It also has a large metal water bowl, which goes at the bottom. The water helps to keep the meat moist, and helps regulate the temperature.

However… The water bowl also turns out to be my new favourite bean-cooking vessel! It’s quite huge (probably about 8 litres) and of course it doesn’t consume any of the precious roasting real-estate. Even better, the juices from your cooking meat will drip into the beans, bringing a nice smokey meaty flavour! ❤️

So anyway, I whipped up a recipe last night, and my awesome wife ran the smoker today while I was at work. I came home to the best baked beans I’ve ever tasted.

The following recipe is plenty for a good 6 people, and should provide some leftovers as well. We used a mix of Apple wood and Cherry wood for smoke.

Ingredients

1 cup cooked bacon, chopped into about 2cm squares (optional)
6 cups mixed raw beans*, soaked in clean water overnight and drained
18 cups water
2 fairly large onions, diced large
3 capsicum (any colour), cut to about 2cm squares
2 Tbsp mustard
270ml of tomato sauce (I used Watties)
6 Tbsp dark brown sugar
8 Tbsp cider vinegar
4 Tbsp Worchestershire sauce
1/2 tsp cayenne pepper (possibly a bit less depending on taste)

* I used a mix of about 40% haricot beans, 10% black beans, 30% red kidney beans, and 20% giant lima beans. The black beans were pleasant surprises, but the lima weren’t particularly enlightening - get butter beans instead, if you can find them.

Mix all the ingredients together in your bowl (or a tray), pop it in the BBQ, and fire it up! The beans will absorb some of the water, and some of the water will evaporate to keep your meat hydrated. Your beans should be cooked by the time the sauce has thickened down - ours were ready in about 5 hours. Make sure you don’t cook them dry!

Enjoy!

Mikrotik RouterOS - incorrect SNMP status for idle WLAN interfaces

Sun, 07 Jan 2018 00:00:00 +1300

While trying to monitor the status of one of our wireless access points, I discovered that RouterOS returns an incorrect SNMP status code for the wlan interfaces when there are no clients connected to the interface. This is at least present in RouterOS v6.40.3, on a Mikrotik cAPlite (RBcAPL-2nD).

Specifically, when the wifi interface is up, but has zero connected clients, the OID returns an SNMP value of 2 (“down”). This status is incorrect - the interface isn’t down, it’s merely waiting for an external connection.

The specific OID is 1.3.6.1.2.1.2.2.1.8.2 (iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifOperStatus - interface #2 in my case)

The applicable RFC says, “When ifAdminStatus changes to the up state, the related ifOperStatus should do one of the following: […] Change to the dormant state if and only if the interface is found to be operable, but the interface is waiting for other, external, events to occur before it can transmit or receive packets” (https://www.ietf.org/rfc/rfc2863.txt, section 3.1.13)

I notice this issue was logged on the forums about 6 years ago, at https://forum.mikrotik.com/viewtopic.php?t=51332.

I’ve raised this issue with Mikrotik support, and apparently it’s been added to their to-do list, but I’m not sure it’ll ever be done. In the meantime, we’ve just had to disable up/down monitoring of that specific interface, because it’s so misleading.

When the problem is DNS: FreeNAS and NFS

Sun, 07 Jan 2018 00:00:00 +1300

I discovered a while ago that NFS requires DNS to function correctly: https://redmine.ixsystems.com/issues/4983

That’s somewhat annoying, because when your DNS server goes down (and your backups are stored on your FreeNAS server and accessed over NFS), is precisely the time when you really want your backups to be accessible.

However, turns out it doesn’t aaaaactuallyyyyy need DNS… it needs name resolution. Specifically, FreeNAS just needs to be able to resolve it’s own hostname. Cue the handy-dandy hosts file.

Under Network -> Global Configuration -> Host name data base, add your details:

192.168.88.5 backupserver backupserver.local backupserver.ad.mydomain.com

If you have a secondary IP address, duplicate the above line and replace the IP as appropriate.

Obviously, replace the hostname and your internal domain name with whatever you use. However, be aware that the .local FQDN is actually required, even if you use a different domain name.

Flashing a Samsung S5 G900I back to stock

Tue, 19 Dec 2017 00:00:00 +1300

I got a second-hand Samsung Galaxy S5 for my mum yesterday (a G900I model, from Telstra), and I spent some time getting it ready.

Firstly, I downloaded the most-recent Telstra firmware image. At time of writing, that’s G900IDVU1CQJ2. Telstra appears to be the only carrier still releasing stock firmware for this phone, so I downloaded the Telstra version even though we’re connecting to Spark New Zealand. This means the phone will at least be running the most up-to-date baseband and modem firmware.

Note - if you try this, and find that the recent (international) version doesn’t work properly with your local carriers, download the older (local) package, and flash the modem.bin and the NON-HLOS.bin packages from the local one instead.

Flashing the new firmware required a little thinking. I’m 99% sure that I did this the hard way, but basically:

Download the newest version of Heimdall from Github
Back last year I had a post that mentioned editing CMakeLists.txt - ignore that now.

Build the software (assuming you extracted the zip instead of using git clone), with:

 cd Heimdall-master
 mkdir build
 cd build
 cmake -DDISABLE_FRONTEND=ON -DCMAKE_BUILD_TYPE=Release .. 
 make

Once I’d built Heimdall, I unzipped the firmware into a convenient place, and I examined the PIT file from the phone:

./heimdall print-pit

This descripts the partition table on the phone. In particular, it tells you which files in your firmware package should be uploaded to which partitions. I ended up with the following Heimdall flash line:

./heimdall flash --APNHLOS NON-HLOS.bin --MODEM modem.bin --SBL1 sbl1.mbn --DBI sdi.mbn --ABOOT aboot.mbn --RPM rpm.mbn --TZ tz.mbn --BOOT boot.img --RECOVERY recovery.img --SYSTEM system.img.ext4 --CACHE cache.img.ext4 --HIDDEN hidden.img.ext4

I know I could have only flashed the modem.bin and NON-HLOS.bin files, but I’m not sure what I would have missed by skipping all the other partitions.

After the phone was flashed up to the current Stock firmware, I made sure it booted correctly, and then went ahead and converted the phone to my beloved LineageOS, happy in the knowledge that all the “other bits” of the firmware were all updated.

Making MindTouch's Dekiwiki work on Debian Stretch

Sun, 17 Sep 2017 00:00:00 +1200

We had accumulated a certain amount of technical debt, due to a Wiki solution that was selected a few years ago: Dekiwiki, by MindTouch. Unfortunately a few months after implementation, MindTouch Core (which Dekiwiki builds on) was well-and-truly deprecated back in 2013.

It all happened before my time, but it seems as if Dekiwiki came as a pre-built VMware Appliance, based on Debian Etch (Debian 4.0; released in 2007). After giving our Dekiwiki environment some serious side-eye for a while, I finally decided to get my hands dirty and try to upgrade it.

Jedd, over at Jeddi.org, wrote a really helpful post back in 2015 about his experiences updating the ageing version of Debian that the VM is based on. The whole Dekiwiki image is held together by sticky-tape, but that post is really helpful in getting Etch upgraded to Wheezy.

Here are a couple additional things I’ve had to do (please, please look at Jedd’s post above - he covers a whole lot that I’m not covering here):

Upgrading Etch -> Lenny:

Follow the guide at Jeddi.org above to get to Lenny. I can’t remember much failing for that upgrade step, but I remember something happened.

From Lenny -> Squeeze:

DNS Breaks. You’ll notice that SSH takes aaaages to log into, and other things will start being slow and complaining.

Resolve this by editing the ‘hosts’ line in /etc/nsswitch.conf:

hosts: files dns

Originally this reads hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4

Note I’m not actually sure this happened between Lenny and Squeeze; it may have been from Squeeze to Wheezy. *shrug*

From Squeeze -> Wheezy:

MySQL breaks. When visiting the wiki, you’ll be informed that it couldn’t access the backend, and that the API might be warming up. However, you’ll notice that the MySQL service fails to start. When you run ‘mysqld’ from command line, you’ll see it’s missing a folder.

Resolve this by running mkdir /var/lib/mysql-files ; chown mysql:mysql /var/lib/mysql-files

From Wheezy -> Jessie (upgrade-only):

Apache breaks. You’ll notice Apache starts serving just the contents of /var/www. This is because the Apache config file now looks for *.conf in /etc/apache2/sites-enabled; of course, the dekiwiki config file doesn’t have the .conf ending.

Resolve this by running mv /etc/apache2/sites-enabled/dekiwiki /etc/apache2/sites-enabled/dekiwiki.conf

From Wheezy -> Jessie (dist-upgrade):

Mono breaks. If you upgrade mono, Dekiwiki will fall back into unlicensed mode. The licensing engine will no longer be able to interpret license files correctly. You’ll see the error “Server license validation failed. The license signature is not valid.”

Resolve this by holding back the mono packages before you run a dist-upgrade:

apt-mark hold libapache2-mod-mono libmono-2.0-1 libmono-2.0-dev libmono-accessibility2.0-cil libmono-accessibility4.0-cil libmono-bytefx0.7.6.2-cil libmono-c5-1.1-cil libmono-cairo2.0-cil libmono-cairo4.0-cil libmono-cecil-private-cil libmono-codecontracts4.0-cil libmono-compilerservices-symbolwriter4.0-cil libmono-corlib2.0-cil libmono-corlib4.0-cil libmono-cscompmgd8.0-cil libmono-csharp4.0-cil libmono-custommarshalers4.0-cil libmono-data-tds2.0-cil libmono-data-tds4.0-cil libmono-data2.0-cil libmono-db2-1.0-cil libmono-debugger-soft2.0-cil libmono-debugger-soft4.0-cil libmono-dev libmono-firebirdsql1.7-cil libmono-getoptions2.0-cil libmono-http4.0-cil libmono-i18n-cjk4.0-cil libmono-i18n-mideast4.0-cil libmono-i18n-other4.0-cil libmono-i18n-rare4.0-cil libmono-i18n-west2.0-cil libmono-i18n-west4.0-cil libmono-i18n2.0-cil libmono-i18n4.0-all libmono-i18n4.0-cil libmono-ldap2.0-cil libmono-ldap4.0-cil libmono-management2.0-cil libmono-management4.0-cil libmono-messaging-rabbitmq2.0-cil libmono-messaging-rabbitmq4.0-cil libmono-messaging2.0-cil libmono-messaging4.0-cil libmono-microsoft-build-engine4.0-cil libmono-microsoft-build-framework4.0-cil libmono-microsoft-build-tasks-v4.0-4.0-cil libmono-microsoft-build-utilities-v4.0-4.0-cil libmono-microsoft-build2.0-cil libmono-microsoft-csharp4.0-cil libmono-microsoft-visualc10.0-cil libmono-microsoft-web-infrastructure1.0-cil libmono-microsoft8.0-cil libmono-npgsql2.0-cil libmono-npgsql4.0-cil libmono-opensystem-c4.0-cil libmono-oracle2.0-cil libmono-oracle4.0-cil libmono-peapi2.0-cil libmono-peapi4.0-cil libmono-posix2.0-cil libmono-posix4.0-cil libmono-rabbitmq2.0-cil libmono-rabbitmq4.0-cil libmono-relaxng2.0-cil libmono-relaxng4.0-cil libmono-security2.0-cil libmono-security4.0-cil libmono-sharpzip2.6-cil libmono-sharpzip2.84-cil libmono-sharpzip4.84-cil libmono-simd2.0-cil libmono-simd4.0-cil libmono-sqlite2.0-cil libmono-sqlite4.0-cil libmono-system-componentmodel-composition4.0-cil libmono-system-componentmodel-dataannotations4.0-cil libmono-system-configuration-install4.0-cil libmono-system-configuration4.0-cil libmono-system-core4.0-cil libmono-system-data-datasetextensions4.0-cil libmono-system-data-linq2.0-cil libmono-system-data-linq4.0-cil libmono-system-data-services-client4.0-cil libmono-system-data-services4.0-cil libmono-system-data2.0-cil libmono-system-data4.0-cil libmono-system-design4.0-cil libmono-system-drawing-design4.0-cil libmono-system-drawing4.0-cil libmono-system-dynamic4.0-cil libmono-system-enterpriseservices4.0-cil libmono-system-identitymodel-selectors4.0-cil libmono-system-identitymodel4.0-cil libmono-system-ldap2.0-cil libmono-system-ldap4.0-cil libmono-system-management4.0-cil libmono-system-messaging2.0-cil libmono-system-messaging4.0-cil libmono-system-net4.0-cil libmono-system-numerics4.0-cil libmono-system-runtime-caching4.0-cil libmono-system-runtime-durableinstancing4.0-cil libmono-system-runtime-serialization-formatters-soap4.0-cil libmono-system-runtime-serialization4.0-cil libmono-system-runtime2.0-cil libmono-system-runtime4.0-cil libmono-system-security4.0-cil libmono-system-servicemodel-discovery4.0-cil libmono-system-servicemodel-routing4.0-cil libmono-system-servicemodel-web4.0-cil libmono-system-servicemodel4.0-cil libmono-system-serviceprocess4.0-cil libmono-system-transactions4.0-cil libmono-system-web-abstractions4.0-cil libmono-system-web-applicationservices4.0-cil libmono-system-web-dynamicdata4.0-cil libmono-system-web-extensions-design4.0-cil libmono-system-web-extensions4.0-cil libmono-system-web-mvc1.0-cil libmono-system-web-mvc2.0-cil libmono-system-web-routing4.0-cil libmono-system-web-services4.0-cil libmono-system-web2.0-cil libmono-system-web4.0-cil libmono-system-windows-forms-datavisualization4.0-cil libmono-system-windows-forms4.0-cil libmono-system-xaml4.0-cil libmono-system-xml-linq4.0-cil libmono-system-xml4.0-cil libmono-system2.0-cil libmono-system4.0-cil libmono-tasklets2.0-cil libmono-tasklets4.0-cil libmono-wcf3.0-cil libmono-web4.0-cil libmono-webbrowser2.0-cil libmono-webbrowser4.0-cil libmono-webmatrix-data4.0-cil libmono-windowsbase3.0-cil libmono-windowsbase4.0-cil libmono-winforms2.0-cil libmono0 libmono2.0-cil mono-2.0-devel mono-2.0-gac mono-4.0-gac mono-apache-server2 mono-common mono-csharp-shell mono-dmcs mono-gac mono-gmcs mono-mcs mono-runtime mono-utils mono-xbuild

Note: Some of these can certainly be unheld. I haven’t yet gone through and identified exactly which component is breaking on upgrade. If you put in the effort to determine the culprit, please let me know.

That’s all! You should now have Dekiwiki running on Debian Stretch! I’m planning to upgrade it to Debian Buster in the next week or two.

Compiling Heimdall without installing Qt

Wed, 13 Sep 2017 00:00:00 +1200

During my recent Wrecked-phone Saga, I had some trouble flashing my firmware. My phone was broken at the time, so I couldn’t enable ADB Debugging - therefore adb wasn’t an option. OEM Bootlock was on, so fastboot wasn’t an option. My Recovery bootloader was also broken. I thought my phone was bricked.

That’s until I remembered Heimdall.

Heimdall is specifically a tool for working with the partitions on Samsung phones - other visitors need not apply. It works a treat for my Samsung Galaxy S5 (kltedv) though.

Heimdall, however, is old. The version in the Debian Sid repository is 1.4.1, which dates back to 2015. And when we have old versions, we also have incompatibilites! Such as Heimdall 1.4.1’s incompatibility with newer versions of Samsung devices … such as my precious SGS5.

Happily, Heimdall is open source, so I was able to download and compile a new version that overcomes the problem. And that’s how I found myself staring at the screen and looking at this:

That is Yuck.bat

I really didn’t want to install 55 packages just to build this. The majority of missing packages were required to build the GUI section of the tool, which I didn’t want to use anyway. What if I could … build Heimdall without building the GUI??

And of course, that’s the title of this article, so here’s how to do it!

EDIT 2017-12-19: The stuff below is no longer necessary. For better instructions, have a look at my article about flashing a Samsung G900I back to stock.

Clone the git repository, with git clone https://github.com/Benjamin-Dobell/Heimdall.git
Edit the CMakeLists.txt file: nano Heimdall/CMakeLists.txt

Delete the last four lines of CMakeLists.txt:

 cmake_minimum_required(VERSION 2.8.4)

 set(CMAKE_MODULE_PATH
 ${CMAKE_SOURCE_DIR}/cmake
 ${CMAKE_MODULE_PATH})

 project(Heimdall)

 set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin)

 option(DISABLE_FRONTEND "Disable GUI frontend" OFF)

 add_subdirectory(libpit)
 add_subdirectory(heimdall)

You’re done! Go ahead and build according to the instructions.

When Encryption Attacks!

Tue, 12 Sep 2017 00:00:00 +1200

So… I wrecked my phone last night. :-(

Android’s phone encryption feature has been around for aaaages, so I was confident when I started the encryption process last night. Unfortunately, something went wrong, and last night I discovered that everything was toast.

After a bit of digging, I found someone else with exactly the same problem!

“unable to boot into my phone as it sits at the boot screen. When I try to boot into TWRP, it asks for my password. […] it appears to decrypt the partition and mount, but then while loading, TWRP shows a continuous stream of:

E:Error parsing XML file

errors until it just restarts.”

Unfortunately, nobody responded to my own personal denvercoder9.

Long story short, I lost everything. But for a while, I thought my entire phone was bricked. The phone wasn’t accessible in either Recovery mode (due to the XML parsing catastrophe) or Download mode - it wasn’t accessible either via adb or via fastboot.

Eventually my mate @rendition pointed out that I could just boot into Recovery, then cancel the decryption step and wipe my phone from there. That allowed me to start again from a clean slate.

And I encrypted my phone right away, with no trouble at all!

Firmware update on an APC AP9630 NMC2

Sat, 02 Sep 2017 00:00:00 +1200

I’ve spent a little while working with the APC Network Management Cards now, and firmware updates are a total pain.

The biggest issue is that the UPS power outlets need to be powered off in order to flash the firmware, otherwise there’s a terrifyingly-high chance that the NMC (a ~$700 card) will completely shit itself, and die permanently. Aside from that, I’ve never managed to get updates working properly from the web interface.

Fortunately, the card is hot-pluggable, so if you have a spare UPS hanging around, you can unplug the NMC from each UPS in your fleet, throw it into your spare, and flash the firmware from there. Fast, easy, and simple… as long as you have that all-important spare.

Download the Firmware

First, download the appropriate firmware for your device. At time of writing, the link for the SmartUPS 6.5.0 NMC firmware can be found here, but if it’s been a while you should find a new version at http://www.apc.com/us/en/tools/download/.

It comes down as an EXE file, but use 7-zip to extract the archive somewhere convenient. You only need the three .bin files:

apc_hw05_bootmon_108.bin
apc_hw05_aos_650.bin
apc_hw05_sumx_650.bin

Flashing the Firmware

Flashing the actual firmware is easy, but you do need to be careful of the order. The bootmon package goes first, then the AOS package, and then the sumx package.

Be aware: the NMC will reboot immediately after each file is uploaded. As a result, you need to reconnect after each upload, and reissue the BIN command.

Make sure you know the username and password (default credentials are apc/apc), and that the FTP server is enabled. Then, go ahead and ftp to the NMC on it’s IP address.

$ ftp
ftp> open x.x.x.x
Connected to x.x.x.x.
Name (eric): apc
Password: apc
230 Login successful.
Remote system type is UNIX.
Using text mode to transfer files.
ftp> bin
Using binary mode to transfer files.

Make sure you send the ‘bin’ command. I also use the hash command so I can watch the transfer status, but it’s unnecessary.

Once you’ve connected, go ahead and upload the bootmon file:

ftp> put apc_hw05_bootmon_108.bin
ftp> close

It’ll only take a few seconds to upload the file, and you won’t see any notification from the FTP server. But if you’re paying attention to the card itself, you’ll see the NMC network connection go down, and then start flashing. This is the NMC installing the new firmware package.

Once the connection is up again, your FTP session will be broken. Go ahead and reconnect; remember to reissue the BIN command:

ftp> put apc_hw05_bootmon_108.bin
ftp> close
<...snip...>
ftp> open x.x.x.x
<...snip...>
ftp> bin
Using binary mode to transfer files.

Now upload the AOS file. When the NMC reboots again, go ahead and repeat the cycle to upload the sumx file.

Once the NMC reboots for the third time, you should be done!

Thanks

Thanks to Angela from APC - she’s been around for years, and as far as I can see she’s THE leading customer support person for the entire NMC firmware suite. You can find her post on this process here at the APC forums.

Respecting Amavis' "Banned Extensions" setting

Sun, 13 Aug 2017 00:00:00 +1200

I’ve been dabbling a little bit with iRedMail, mostly just to have a play with a mail server, but also to see what’s involved in mail security. iRedMail is a package that pulls together Postfix as an MTA, Dovecot as a POP3 & IMAP server, SOGo for ActiveSync, Roundcube for Webmail, SpamAssassin for spam protection, and ClamAV for virus scanning.

Okay I have no idea why I have to write this, but apparently it’s a thing.

Amavis has a list of banned file extensions. In Debian, they live in /etc/amavis/conf.d/20-debian_defaults, and /etc/amavis/conf.d/50-user, and are set in the $banned_filename_re variable. THIS MAKES PERFECT SENSE.

But of course, there’s always something that doesn’t make sense, and that is the fact that there is a SQL backend (at least in the environment created by iRedMail), and settings in here take precedence over the Amavis config files somehow.

And even more bizarrely, there exists in this SQL environment, a policy setting entitled “banned_files_lover”, which was set to “Y”. I shit you not. My only hope is that this only defaults to “Y” for postmaster.

To fix this, you need to hop into the database, and update the appropriate column in the policy table:

$ mariadb
MariaDB [none]> \u amavis
MariaDB [amavisd]> update policy set banned_files_lover="N";

And now, Amavis will obey your file extension filters!

Thankfully, I found this information at http://www.iredmail.org/forum/topic13147-iredmail-support-amavisd-passed-but-setup-at-ddiscard.html – I never would have found it otherwise!

Tuna Patties

Sun, 13 Aug 2017 00:00:00 +1200

What do you do when it’s dinner time, and you haven’t got any ingredients ready for the meal?

What if (perhaps as a result of being an ultralight hiker), you also have a huge surplus 500g sachet of tuna?

Tonight these particular stars aligned, and I ended up making these amazing Tuna Patties: http://allrecipes.com.au/recipe/15705/tuna-patties.aspx

I did tweak the recipe considerably, though:

Ingredients:

2-3 large potatoes, peeled and cubed for mashing
Optionally, swap one of the potatoes for a kumara or some pumpkin
500g tuna flakes
2 carrots, grated and excess juice dabbed out with a paper towel
A goodly handful of frozen chopped spinach
1 onion, cubed
3 garlic cloves, chopped
1/8th tsp cayenne pepper
1/2 tsp basil
1/2 tsp oregano
1/2 tsp sage
1 tsp veggie stock powder
2 eggs
2 tbsp mustard

Method:

Boil the potatoes in salted water until cooked.
In another pan, fry the onions for 2 minutes, then add garlic.
Fry until the onions start to brown, then add herbs, spices, and stock powder.
Continue to fry for another minute, then remove from heat and let cool.
When the potatoes are cooked, tip out all the water, and allow remaning water to evaporate.
Mash the potatoes - they’ll seem unpleasantly dry, but that’s OK.
Mix the carrots, mashed potatoes, fried onion and garlic, all together in a bowl.
In another bowl, mix together the tuna, the eggs, and the mustard.
Now combine everything together, and give it a good mix.
Form the mixture into patties and fry in a lightly oiled pan.

In retrospect, this recipe seems really difficult, but it was actually quite straightforward, and it was totally delicious. Definitely good with a bit of mayo on the patties. Yummo!

Making Amavis work with ESET Antivirus

Sat, 12 Aug 2017 00:00:00 +1200

But of course, ClamAV has shown disappointing performance, and it would be really nice to use something more… commercially suitable.

To tie together mail receipt and scanning, iRedMail uses Amavis (strictly speaking, ‘amavisd-new’). Amavis uses ClamAV by default, but it comes with a bunch of configuration blocks to bring together other antivirus applications.

But although amavisd-new is stable and still maintained, some parts of it are really old. In particular, many of these av-scanner config blocks are… uhh… “deprecated”. There’s one particular entry for ESET that is dated 2002 - things have changed a lot in the last fifteen years. *shudder*

So, with the help of some documentation, I managed to piece together a code block that works:

['ESET File Security for Linux',
 ['/opt/eset/esets/sbin/esets_scan','esets_scan'],
 '--subdir --unsafe --unwanted --clean-mode=strict {}',
 [0,10,100],[1,50],
 qr/threat="([^"]+)"/m
],

That would be the end of the story, except I also had to hunt around to find out the right place to put my beautiful codeblock into. It turns out that Debian’s Amavis config structure is quite different to the CentOS config that is most-frequently mentioned in the iRedMail forums. I spent a lot of time playing with /etc/amavis/conf.d/15-av_scanners, and nothing seemed to work. Eventually I found out that Debian features a /etc/amavis/conf.d/50-users file that overwrites the settings from 15-av_scanners. Finally I had progress!

Somewhere around line 154 in /etc/amavis/conf.d/50-users, you’ll find an @av_scanners codeblock. I deleted the ClamAV section in there, and replaced it with the ESET codeblock above. I left the ClamAV settings in the @av_scanners_backup section, because Amavis will fall back to that if ESET fails.

That seems to be all! At least, it works with the EICAR anti-malware test file.

RingZer0team CTF - Challenges 86, 87, and 88

Thu, 13 Jul 2017 00:00:00 +1200

There are a bunch of fantastic Capture The Flag security challenges on RingZer0Team.com. I’ve been working through some of these for a wee while now, and with the New Zealand Cyber Security Challenge coming up again soon, I thought I’d get back into some of them.

Challenge 86 (“1/3 Do not waste the environment”, under the Forsensic Challenges) is one of a series of challenges where you need to dig through some provided data to find the flag.

I started by downloading the ‘forensic bundle’, which was just a large zip file. The first challenge was to figure out the contents of that zip file. The file name was just a jumble of characters, and there was no extension. Running file on it just returned the file type “Data”… not very enlightening, but head uncovered the string “VBOX” in the file… okay, it’s VirtualBox file.

Because I didn’t have Virtualbox installed, I spent a bit of time digging through the actual Vbox file itself, trying to see if there’s a flag already in there. I used strings | egrep -i flag-...., to try to find any flags embedded in the file itself. I didn’t find any flags, but I did discover that the computer name itself is “FLAG-PC”. Very clever. :-|

Update, many hours later:

Okay this totally cooked my bacon. After hours of playing with VirtualBox and playing with files, I finally gave up and Googled the name of the downloaded file.

Turns out I was REALLY close. The flag IS embedded in the actual file itself, but the format is different for the first time. The correct grep string would have been egrep -i '.?f.?l.?a.?g.?-'.

Real frustrated by how close I was, I’ll have to expand my searches in the future. Hat-tip to @professormahi and his GitHub page.

Once I understood that, I also discovered the flag for challenge 87 (“2/3 Did you see my desktop?”), in the same manner.

I haven’t managed to get the flag for Challenge 88 (“3/3 Suspicious Account Password”) yet, but I do know what it is. I can see (by grepping for “Visited”) a bunch of visits to http://www.forensicswiki.org/wiki/Tools:Memory_Imaging… this thing is a memory dump. That explains now why it doesn’t work as a VirtualBox disk.

Next time I sit down here, I’ll find visit Forensics Wiki, and start digging through the memory dump for the passwords.

Update two

I decided not to put this up as a separate post, because once I knew what I was looking at, it was pretty easy. Again though, I relied on @professormahi’s work - but now thanks to him I have a little bit of experience with Volatility!

First, we get Volatility to scan the image and see what it’s dealing with:

$ volatility -f vm.vmdk imageinfo
Volatility Foundation Volatility Framework 2.6
INFO : volatility.debug : Determining profile based on KDBG search...
 Suggested Profile(s) : Win7SP1x86_23418, Win7SP0x86, Win7SP1x86
 AS Layer1 : IA32PagedMemory (Kernel AS)
 AS Layer2 : FileAddressSpace (/home/eric/Downloads/86/vm.vmdk)
 PAE type : No PAE
 DTB : 0x185000L
 KDBG : 0x82920be8L
 Number of Processors : 1
 Image Type (Service Pack) : 0
 KPCR for CPU 0 : 0x82921c00L
 KUSER_SHARED_DATA : 0xffdf0000L
 Image date and time : 2014-03-09 20:57:55 UTC+0000
 Image local date and time : 2014-03-09 13:57:55 -0700

Now, knowing the profile that we need to use, we can use it to just… dump the password table:

$ volatility -f vm.vmdk --profile Win7SP1x86 hashdump
Volatility Foundation Volatility Framework 2.6
Administrator:500:aad3b435b51404eeaad3b435b51404ee:31d6cfe0d16ae931b73c59d7e0c089c0:::
Guest:501:aad3b435b51404eeaad3b435b51404ee:31d6cfe0d16ae931b73c59d7e0c089c0:::
flag:1000:aad3b435b51404eeaad3b435b51404ee:3008c87294511142799dca1191e69a0f:::

And then, we grab that NTLM hash and pop it into an online NTLM decryptor service, and we’re away!

Staying Secure in Business

Wed, 28 Jun 2017 00:00:00 +1200

A friend asked a question today on Facebook. I started writing a reply, and it turned into a 700-word essay. Hate it when that happens…

The question was:

“If large multi-international companies are getting hacked in Europe who have millions if not billions of $$ and capacity to protect their IT systems - how can small/micro businesses protect their IT platforms and systems?”

Firstly - nobody is safe from an APT (“Advanced Persistent Threat”). The recent NotPetya outbreak was an APT - malicious actors hacked a Ukrainian firm that produced accounting software, and used their software’s update facility to literally deliver malicious code specifically to the users of this particular software package (“M.E. Doc”). Nothing’s keeping folks safe from that level of sophistication.

However, the exploit by which NotPetya spreads was actually patched by Microsoft in March. If people had updated their internet-connected devices, this virus would have been limited to only users of M.E. Doc.

Aside from that though, there’s actually a disappointingly simple answer to this. Security is actually relatively within reach, if we all follow these simple steps:

1) Update. Everything. At every opportunity.

That includes Windows Updates, but also Adobe Reader (annoying!), Java (double annoying!), anti-virus engines, etc. This is absolutely the top priority, and the cybersecurity industry has been trotting this line out for years, but people just don’t. Because annoying.

Real life incident: June 22, 2017 - Honda falls prey to the WannaCry worm, 37 days after release (four months after the patch was released), due to poor patching hygiene.

See http://thehackernews.com/2017/06/honda-wannacry-attack.html

2) Backup everything important.

The more important it is, the more places you should keep it. You should have at least three copies of every important file. THIS IS ANNOYING, but if you don’t do it, data loss will hit you. Something like Crashplan (a low-cost service that sends backups to the cloud), plus a regular (weekly?) copy to a read-only media (e.g. a DVD), should be enough for a small business.

Real life incident: Feb 1, 2017 - GitLabs suffers a major outage after data was accidentally deleted, followed by a sequence of discoveries that none of their five layers of backup or replication techniques “are working reliably or set up in the first place”.

See https://techcrunch.com/2017/02/01/gitlab-suffers-major-backup-failure-after-data-deletion-incident/

3) Use next-generation anti-virus software.

Many of the AV products these days include very clever ways of analysing the behaviour of unknown software, and literally just rolling back all the changes if the AV decides it’s not trustworthy. Webroot is a good example of this, at a very affordable price.

Real life incident: April 23, 2017 - the Adylkuzz virus started quietly spreading around the internet, and setting up Monero cryptocurrency mining operations on infected PC’s. Because the payload wasn’t destructive, it flew under the radar and was missed by many antiviruses for weeks.

See https://www.proofpoint.com/us/threat-insight/post/adylkuzz-cryptocurrency-mining-malware-spreading-for-weeks-via-eternalblue-doublepulsar

4) Be alert, cautious, and sufficiently paranoid.

The last - and hardest - step is to maintain a healthy sense of distrust; sometimes referred to as “sufficient paranoia”. When you receive an email, ask yourself: is this email in-character for this person? Is the spelling what I’d expect? Is it appropriate? Is it asking me to take an action (even if that action is just opening a file)? If so, is it expected, or is it out of the blue? If it’s out of the blue, why? What’s the story behind that?

Real life incident: Sept 30, 2011 - I received an email from a lady I’d once interviewed for a job. It was a distraught email saying she was stuck in Wales, had been mugged, lost all her money, and needed a brief loan until she got home. I wrote her an email (not a reply), telling her email had been hacked; lo and behold, I got a reply! “It’s me… this is for real, i have checked with the consulate but there is nothing really working out, most important is i don’t have enough money on please, please i need you to loan me some, i can pay you back once i get home.”

I ended up finding her cellphone number and sending her a message - she was at home, safe and sound.

See https://blog.malwarebytes.com/cybercrime/2014/08/email-hijack-leads-to-i-was-robbed-send-me-money-scam/ – this was exactly the wording I received.

Geez, that was a long Facebook post Eric.

It really was, and a few minutes afterwards, this tweet came to my attention:

https://twitter.com/josephfcox/status/879982828418719744

It’s fun to laugh, but it’s a good reminder that there’s always going to be a weakness. You need to structure your business disaster recovery in a way that you can quickly and efficiently deal with a complete wipe-out like this.

And we’re done.

So, to summarise this essay, you need to:

Always install updates
Always take backups
Use AV software with full journaling and rollback capabilities
Practice sufficient paranoia

With these four steps, you will be as safe as the biggest corporate. (roughly speaking)

RingZer0team CTF - Challenge 44

Sat, 24 Jun 2017 00:00:00 +1200

Challenge 56 (“Hey Chuck where is the flag?”, under the Forensic Challenges) started by only offering a .pcap file. I downloaded Wireshark and had a quick dig. The packet capture consisted of a brief browse of a “Chuck Norris Facts” website.

After quickly flicking through the pcap itself looking for low-hanging fruit, I used File -> Export Objects -> HTTP. This takes all the HTTP traffic, recombines the packets into files, and saves them.

Once I’d done that, I opened up the files and had a quick flick through. I checked the images for metadata etc but didn’t find anything. However I did fine a picture of Chuck Norris, appearing to urge the reader to keep searching files – in French.

When I didn’t find anything super-obvious in the image contents, I hopped into Bash and just ran grep -i flag *. It quickly returned the flag, stored in a php file!

Nice and easy hunt, great way to finish the day.

RingZer0team CTF - Challenge 65

Sat, 24 Jun 2017 00:00:00 +1200

This is a continuation of my series on RingZer0Team.com.

Challenge 65 (“Hide my ass in my home”, under the Forensic challenges) is a nice quick little forensic challenge. You get to download a .tar file, and do a bit of digging to uncover the flag.

This was a nice easy few minutes. After downloading the archive, I discovered a few files inside - including a particularly groovy electronica swing track, which I’m listening to again right now. Yeaaaaahhhhh.

Apart from a groovy mp3, the archive contains some dot-files such as .bash_history, a .jpg picture, an index.html, and some other bits.

I found the Flag in one of the dot-files; specifically, it was inside a Vim swap file that seemed to be leftover from a power failure or similar.

WireGuard - Part Three (Troubleshooting)

Mon, 12 Jun 2017 00:00:00 +1200

This is part of my brief series on WireGuard. I’m pretty enamoured with WireGuard and the way it works, but there were a couple slightly curly bits that I needed to get my head around. This troubleshooting guide is a rough dump of the issues I had, and how I fixed them.

Gotten Stuck?

At this stage, there are actually a few ways that this can go wrong, even though we haven’t done much. Think through all the bits:

Installed WireGuard at both ends
Set up your NAT rule on the remote side
Created a private and public key on each side
Put each public key in the opposite side’s [Peer] config
Put a suitable IP address on each side

Troubleshooting

There are, actually, a myriad of ways this can fail:

Broken routing on the client PC
Broken NAT on the remote router
Broken routing on the remote PC
Broken wg0 configuration on one side
Lack of TCP forwarding on the remote computer
Lack of Proxy ARP on the remote computer

If you can’t ping the remote server yet, don’t panic. Run tcpdump to find out what you’re missing.

On the remote server: tcpdump -i wg0
On your local machine: ping -c1 10.20.40.1

That will tell you whether your packets are reaching the remote server, or if they’re not getting through the tunnel.

If they’re not making it through the tunnel at all, you’ll probably be seeing error messages in the ping. For example:

PING 10.20.10.1 (10.20.10.1) 56(84) bytes of data.
From 10.20.40.2 icmp_seq=1 Destination Host Unreachable
ping: sendmsg: Required key not available

That’s the error message I got when I set AllowedIPs too strictly. Because I was trying to ping something that was routable, but wasn’t within the AllowedIPs range, there was no applicable key for the packet.

Thanks

Huge gratitude to Jason Donenfeld (aka zx2c4) for spending his time not only reading this post, but also for sending me some fantastic feedback! I’d made some bungles in my original post on this topic, and he vastly helped my understanding.

Also, huge gratitude to another Jason (@rendition) who has helped me develop from a junior network admin into a … ‘moderate’ network admin. I’ve learned more in the last year than I ever thought possible. He’s taught me nearly everything I know about managed networking, reviewed this post for me, and is actually the guy who introduced me to WireGuard originally!

WireGuard - Part Two (VPN routing)

Sun, 11 Jun 2017 00:00:00 +1200

This is a continuation of my brief series on the new WireGuard VPN. Part One was about the simple building-blocks to get WireGuard working between two endpoints. Now that we’ve got a couple machines able to ping each other by IP address, we can carry on a bit deeper into the inter-LAN routing stuff.

Extending on from the IP addresses in Part One, instead of JUST connecting to the remote machine, I want to actually have access to everything on the whole 10.20.0.0/16 network; even the non-WireGuard devices. I want it to be like I’m there on-site. The idea here is roughly:

Remote machine: wg0: 10.20.40.1 (behind public IP 163.172.161.0)
Local machine: wg0: 10.20.40.2 (with eth0 on 192.168.88.207)
Machines on remote LAN: 10.20.0.0/16
Remote LAN Watchguard range: 10.20.40.0/16 – note this is within the remote 10.20.0.0/16 range

From my local machine, with minimal interaction, I want to be able to ping something like 10.20.10.30, and get a response.

Happily, now that the two endpoints are talking with each other, there’s really not much that has to happen to get things working the way I want.

Remote Config

Many Linux distros disable IPv4 packet forwarding by default. But in this case, we very much want this enabled. Edit your /etc/sysctl.conf file, and look for the line: net.ipv4.ip_forward, and set the value to 1. You may need to add this line manually, or uncomment it.
You’ll also need to enable ProxyARP. This is another kernel setting that’s usually disabled, so edit /etc/sysctl.conf, and add the following line: net.ipv4.conf.all.proxy_arp = 1.
Reload your kernel settings. A reboot will do the trick, or you could just run sudo sysctl -p /etc/sysctl.conf.

Starting WireGuard on Boot

There are a variety options to do this, and if you’re a seasoned sysadmin you probably already have a favourite way.

One option is to add post-up wg-quick up wg0 to the tail of your eth0 block (or appropriate interface) in /etc/network/interfaces.

A second option (for systemd users) is to simply run systemctl enable wg-quick@wg0. This will tell systemd to bring the wg0 interface up once a network connection is established.

I only do this on the remote machine, because I don’t want my local machine to be forever spinning up it’s WireGuard connection; however I always want the remote machine listening for my packets. That said, there’s probably no reason to avoid this; I just haven’t done it yet.

Local Config

Here’s the really cool bit. Since your remote machine is now set up to perform IPv4 forwaring and ProxyARP, the only thing you need to change on the local side is a single number. Or rather, a couple numbers.

In your /etc/WireGuard/wg0.conf file, just expand the network range of your interface to include the entire remote LAN:
```
 [Interface]
 ...
 Address = 10.20.40.2/16

 [Peer]
 ...
 AllowedIPs = 10.20.0.0/16
```
Now reload your WireGuard config, either by rebooting, or running sudo wg-quick down wg0 && sudo wg-quick up wg0.

That’s all! When you make the above changes, wg-quick will modify your routing table so that any IP within 10.20.x.y will be sent over the wg0 interface. The remote host will dutifully forward the packet out into the rest of the network, and ProxyARP will take care of the rest:

$ ping -c1 10.20.10.31
PING 10.20.10.31 (10.20.10.31) 56(84) bytes of data.
64 bytes from 10.20.10.31: icmp_seq=1 ttl=63 time=91.8 ms

Troubleshooting

Okay so by now there’s a really good chance that you’ve bumped into trouble. I’ve started on a basic WireGuard troubleshooting guide, but it only covers the issues that I bumped into. If you’re still stuck after reading through that, ask a question on the WireGuard Mailing List, or reach out via IRC (#WireGuard on Freenode) - this is all on the WireGuard website.

Wrap-up

That should be all! When you reboot the remote server (I specify reboot, because you want to test that your solution can survive an outage at the remote site), you should be able to ping other things in the remote LAN without any additional interaction. I still manually run wg-quick up wg0 on my local laptop, because I don’t want to be connected remotely all the time.

Thanks

If you do end up using WireGuard, go forth and donate! Seriously, at very least, send Jason the cost of a local cup of coffee or a beer for his efforts.

WireGuard - Part One (Installation)

Sat, 03 Jun 2017 00:00:00 +1200

WireGuard is the most excellent VPN stack around. It’s really fast, the concept of Cryptokey Routing is awesome, and I love the speed and simplicity benefits that come from opionionated cryptography. The protocol is so simple - expressed in a mere 4k lines of code - that it’s auditable by anyone.

But.

With my initial naive approach, I found myself using HTTPS, over ports forwarded over SSH tunnels, connected over WireGuard. Although it was straightforward to get WireGuard working between two endpoints, I ended up in nested-crypto hell.

So, this brief series is about making WireGuard work as a VPN onto a different site. We’ll start by making it work between two endpoints, and we’ll go forward from there. My end goal is to have access to all the resources on a remote site, just by running wg-quick up wg0.

The endpoints I’m going to set up are at:

Remote machine: wg0: 10.20.40.1 (behind public IP 163.172.161.0)
Local machine: wg0: 10.20.40.2

NAT Setup

Chances are, your remote endpoint is behind a firewall of some sort. Pick a high port, and configure your firewall to forward UDP packets on that port through to your remote WireGuard endpoint. You don’t need to do this on your local side, because reply traffic from the remote side will generally be handled by the stateful session part of your firewall.

Many routers and firewalls offer port address translation (also known as PAT) as part of port forwarding or NAT. This is when a packet hits the firewall on (say) port 57432, and the firewall puts it on the LAN to port 22, for example. You don’t want this. I think it’s possible with WireGuard, but it adds complexity without benefit.

Config - Remote Site

Install WireGuard

Generate your keys. The following will create a public key and a stub config in /etc/WireGuard/:

 cd /etc/WireGuard
 umask 077
 printf "[Interface]\nPrivateKey = " > wg0.conf
 wg genkey | tee -a wg0.conf | wg pubkey > publickey

Edit your config to match:

 [Interface]
 PrivateKey = WhAt3v3R= (this is the private key generated on this machine)
 ListenPort = 12345 (this is the UDP port you've forwarded from your firewall)
 Address = 10.20.40.1/24 (this will be the IP given to the wg0 interface)

 [Peer]
 PublicKey = (leave this blank for now; you'll paste in your local public key here soon)
 AllowedIPs = 10.20.40.0/24 (this is the range of WireGuard IP addresses that this Peer's key can be used from)

That should be all you need to configure on the remote side for now. Save your wg0.conf file, and bring the interface up:
```
 wg-quick up wg0
 ping -c1 10.20.40.1
```
You should see a bunch of actions performed by wg-quick, and a reply packet from your ping. Now onto the local side.

Config - Local Machine

Install WireGuard and generate your keys, as per the first two steps above.

Edit your configuration again:

 [Interface]
 PrivateKey = WhAt3v3R+PaRt-tw0= (this is the private key generated on this machine)
 ListenPort = 12345 (this is the UDP port again; I don't think they have to be the same)
 Address = 10.20.40.2/24 (note - different IP address, but in the same range)

 [Peer]
 PublicKey = ?????????? (copy the public key from the REMOTE server here)
 Endpoint = 163.172.161.0:12345 (enter the PUBLIC IP address of the remote site, plus the forwarded port)
 AllowedIPs = 10.20.40.0/24 (specifying that packets using this key must come from within 10.20.40.x)

Now that you’ve got a local public key, take a second to paste it into the Peer public key section on the remote server.
Once you’re finished you should be able to bring the interface up and ping it:
```
 wg-quick up wg0
 ping -c1 10.20.40.2
```
… and you should even be able to ping the remote server as well:
```
 ping -c1 10.20.40.1
```

Gotten Stuck?

At this stage, there are actually a few ways that this can go wrong, even though we haven’t done much. Here’s a quick summary of everything we’ve done:

Installed WireGuard at both ends
Set up your NAT rule on the remote side
Created a private and public key on each side
Put each public key in the opposite side’s [Peer] config
Put a suitable IP address on each side

If you’ve nailed each of those and you’re still having trouble, you can have a quick look at the brief Troubleshooting guide I’ve put together as Part Three.

Onto Part Two

That should be all you need to get WireGuard working between two machines on two different sites. So far we haven’t done anything either interesting or uncommon - this is all the basic stuff you’ll find on the WireGuard Quick Start page, although expressed slightly differently. The next article will be a bit more about intra-site routing.

Thanks

If you do end up using WireGuard, go forth and donate! Seriously, at very least, send Jason the cost of a local cup of coffee or a beer for his efforts.

Pan-fried Spicy Miso Chicken

Mon, 29 May 2017 00:00:00 +1200

Jeez I just discovered I’ve only posted a single recipe on this. Time to change that.

Chicken for dinner tonight. I was feeling a little adventurous, and definitely wanted some miso involved, but also felt like cayenne pepper and shallow-fried chicken strips. Here’s the result! This went really well on a base of bulgur wheat, and with a side of stir fried veggies.

Ingredients

Two chicken breasts, skinless
2 tsp miso paste
2/3 cup plain flour
2 tsp celery salt
1 1/2 tsp onion powder
2 tsp smoked paprika
1 tsp chicken stock powder
1/8 tsp cayenne pepper
1/2 tsp black pepper

Method

Cut the chicken into strips, sized similarly to your two thumbs together.
Sift all the dry ingredients into a bowl, and stir or shake until mixed.
Taste your flour coating!! Make sure it’s delicious, and make sure it’s spicy enough.
Mix your miso paste with a little hot water until smooth. It should be quite thick, you don’t want it dripping off.
Coat all the chicken in the miso paste.
Roll the chicken strips in the flour until thoroughly coated.
Pan-fry in a very hot pan, for about 2 minutes per side.

Enjoy!

RingZer0team CTF - Linux Sysadmin challenges

Sun, 28 May 2017 00:00:00 +1200

The Sysadmin Linux series of challenges is where you’re trying to breach the security of a Linux system. I actually finished most of these last year, but I wanted to finish my last two. Of course, to get to the last two stages, you need to use the flags from the previous stages. So I’m revisiting them.

Level 1 - Morpheus -> Trinity

We start by SSH’ing into a particular user account on the ringzer0team server:

You have mail.
Last login: Thu Apr 27 02:52:40 2017 from <somewhere>
morpheus@forensics:~$

There’s only one file in the home folder, and I can’t read it. There’s a /home/trinity folder with full read access, but also nothing legible.

The flag for level 1 is found by running ps aux, which reveals what appears to be Trinity’s password:

root 3241 0.0 0.0 4188 572 ? S Jan14 1:44 /bin/sh /root/backup.sh -u trinity -p Flag-<redacted>

Level 2 - Morpheus -> Architect

Aha, and the flag for level two is in /etc/fstab, which contains what appears to be The Architect’s password, in base64:

/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0
/dev/fd0 /media/floppy0 auto rw,user,noauto 0 0
#//TheMAtrix/phone /media/Matrix cifs username=architect,password=$(base64 -d "<redacted>"),iocharset=utf8,sec=ntlm 0 0

Level 3 - The Architect

Level 3 requires you to log on as architect, using the password we discovered earlier. The only clue is “dig for password”.

This bit got a bit harder. There are a couple possible hints in architect’s mail file, including a bunch of attempts to mount a cifs volume, and reference to a file in /backup/.

Digging through the files in /backup left me at a loss. There are references to /tmp/Gathering.py, but that’s a challenge for later (cypher’s challenge). After some digging I located /var/tmp/.swl, which contained some MySQL credentials!

mysql> show tables;
+----------------+
| Tables_in_arch |
+----------------+
| arch |
| flag |
+----------------+
2 rows in set (0.00 sec)

mysql> select * from arch;
+------+-----------------------+
| id | arch |
+------+-----------------------+
| 1 | The one |
| 1 | The null one |
| 1 | The mother of the one |
| 1 | The father of the one |
| 1 | The flag of the one |
| 1 | The null one |
+------+-----------------------+
6 rows in set (0.00 sec) 

mysql> select * from flag;
+---------------------------------+
| flag |
+---------------------------------+
| FLAG-<redacted> |
+---------------------------------+
1 row in set (0.00 sec)

Yasss after a solid hour of digging, I’ve found the flag for the third level!

Level Four - Morpheus -> Oracle

Again we’re logging on as morpheus, but this time the goal is to access the oracle account.

Lots of tasty stuff under /var/tmp/.viminfo, now that I’ve found it. References to files in /backup/, to /tmp/Gathering.py, /tmp/mail_cypher

Don’t make the mistake of trying to cat .swo; it’ll eat your console session.

Found this interesting thing in mail, but it’s not helpful: forensics.localdomain : Apr 14 21:14:33 : morpheus : user NOT in sudoers ; TTY=pts/4 ; PWD=/home/trinity ; USER=root ; COMMAND=/usr/sbin/usermod -aG neo morpheus

In one of the /backup files, I found the remains of an SSH key, and it worked! Access to the oracle account achieved. In the home folder, we’ve got a handy-dandy Base64-encoded flag in flag.txt.

Level Five - Oracle Encrypted File

There’s a file in the oracle home folder named encflag.txt.enc:

U2FsdGVkX1+dCl4WEHNJKBqA8a4fQeheOgA7oiNmjwlJQvGaQAgqcIsGRIcbdHKF
heSs51JRSEmOLqVyGvoxDA--

Decoding the base64 returns a binary blob, prefixed with the string “Salted”. This is one of the levels that I never completed originally, so maybe it’s time to give it another crack.

A quick google of “salted base64” reveals an interesting comment on StackExchange. Apparently I’m looking at a file encrypted with the OpenSSL ’enc’ command, so I first copy the file to my own computer to work.

I discover that the command openssl enc -in testfile -d -a will un-base64 it, then load the file in. I need to pass it a password to decrypt with though, so now I need to find that.

Oh my god I just did cat .* in the oracle home folder and literally found an alias that reads and decrypts the file. I can’t believe that took me almost an hour.

There are no words.

Level Six - Trinity -> Neo

Okay getting serious now. Using Trinity’s account, I need to find the password for the neo account.

It’s a bit noisy on login:

You have mail.
Last login: Tue Apr 25 16:29:29 2017 from <le blah>
-bash: hello: command not found
Sup Neo!
ls: cannot open directory /home/neo: Permission denied
cat: phonebook: Permission denied

A vague tickle in my memory prompted me to run sudo -l right off the bat. Rewarded with:

User trinity may run the following commands on this host:
 (neo) /bin/cat /home/trinity/*

So of course I immediately run sudo -u neo /bin/cat /home/trinity/*

This gives us a ‘phone book’ containing:

The Oracle 1800-133-7133
Persephone 345-555-1244





copy made by Cypher copy utility on /home/neo/phonebook

Trinity’s .bashrc contains a bit of kruft:

hello neo
echo "Sup Neo!"
$(ls -lart /home/neo)
cat phonebook

Really not useful. I remember finding Trinity’s password with ps aux:

root 3241 0.0 0.0 4188 572 ? S Jan14 1:47 /bin/sh /root/backup.sh -u trinity -p Flag-08grILsn3ekqhDK7cKBV6ka8B

It looks a little bit like a MySQL connection string, so I try that but no dice. Nothing under /var/tmp this time, and grep doesn’t find anything useful under the /backup goldmine.

There’s some weirdness under /etc/passwd. I don’t know what this means, but saving it for later:

trinity:x:1002:1002:trinity,%,lsdf(940998+(n.~,3):/home/trinity:/bin/bash

I looked through the mail file, but nothing jumped out at me there. Also trawled again through /backup, as well as /var/backups. Then /etc and /var. Argh.

Finally I start looking for Persephone. Nothing helpful, but it reminds me of the last line in the phonebook.

copy made by Cypher copy utility on /home/neo/phonebook

And then I remember that the sudo command allows a * at the end of the cat command, which might allow me to do path traversal…

sudo -u neo /bin/cat /home/trinity/../neo/phonebook

Finally, after a whole hour of digging, I’m awarded Neo’s password!

Level 7 - Neo is Not Alone

I don’t know what this clue means, but I bet it has something to do with Persephone. Persephone comes from Greek mythology as one of Zeus’ daughters, and the goddess of the underworld. Does that help me here? I don’t know, it’s getting late…

I run cat .* in Neo’s home directory to see if there’s anything hiding there, but no luck this time. Also nothing for sudo -l.

There’s an unreadable file (owned by root) called ‘result.stat’, that’s probably related to something. I browse the mail history and see:

forensics.localdomain : Dec 7 14:08:08 : neo : user NOT in sudoers ; TTY=pts/2 ; PWD=/home/neo ; USER=morpheus ; COMMAND=/home/morpheus/egrep /home/morpheus/../neo/result.stat

That could be handy. I do remember an egrep in /home/neo, will have a look. Lots of reference to /bin/monitor, too.

I’m only in the neo group, so nothing to see there.

Running ps aux | grep neo shows a heap of /bin/monitor processes started by root, but running under neo. There’s also a “SCREEN” instance there, but running screen -D -RR doesn’t reclaim it, so something’s up there. I’m pretty sure there’s something there in /bin/monitor.

In all honesty this one had me beat. It was half-past midnight and I was exhausted. I knew that there was something special about /bin/monitor, and I knew there was something special about the fact that it was running as neo. But I took the expedient route and had a quick google. I found the answer at http://blog.dornea.nu/2016/10/30/ringzer0-ctf-sysadmin-linux/ - without this I could have been looking for years. However, as is so often the case, this tutorial was the basis of some great learning for me.

Because the process is owned by neo, and because /bin/monitor could be read by neo, it meant I could run a stack trace. I have never done this before - no false impressions here, I totally relied on that dornea.ru page to tip me off to the usage of strace. But once I knew that, I had the flag.

I was very well misled by this one, actually. The Persephone comment in the phonebook totally led me down the wrong path, and I spent quite a long time searching for her details on the system. It was the reddest herring possible.

Level 8 - Morpheus -> Cypher

Okay it’s late now so I’m going to try to thrash this one quick. Nothing in ps aux for cypher. Grepping /backup found a python file and a dump showing a cron job runs this file every minute. The contents of the python file are:

import os
os.system('ps aux > /tmp/28JNvE05KBltE8S7o2xu')

I can’t read the output file, but I can edit the python script! This is gonna be easy.

After a little bit of trial and error, I edited the python script to first list the contents of the /home/cypher folder, and after that, to output the contents of /home/cypher/flag.txt

It’s important to note that the python file is recreated every three minutes when it’s run, so you need to redo your changes for every iteration.

Fin

That’s the end! I’m quite pleased I’ve managed to finish all of the Sysadmin Linux section. Of course that’s also scary, because it means I need to break into the other sections. That’s OK though… it’s a good time to do it!

RingZer0team CTF - Challenge 148

Sat, 27 May 2017 00:00:00 +1200

Challenge 148 (“Sysadmin Linux Level 2”) is one of a series of challenges where you’re trying to breach the security of a Linux system. I actually finished most of these last year, but I wanted to finish my last two. Of course, to get to the last two stages, you need to use the flags from the previous stages. So I’m revisiting them.

We start by SSH’ing into a particular user account on the ringzer0team server:

You have mail. Last login: Thu Apr 27 02:52:40 2017 from morpheus@forensics:~$

There’s only one file in the home folder, and I can’t read it. There’s a /home/trinity folder with full read access, but also nothing legible.

Running ps aux reveals what appears to be Trinity’s password:

root 3241 0.0 0.0 4188 572 ? S Jan14 1:44 /bin/sh /root/backup.sh -u trinity -p Flag-

Aha, and /etc/fstab contains what appears to be The Architect’s password, in base64:

/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0 /dev/fd0 /media/floppy0 auto rw,user,noauto 0 0 #//TheMAtrix/phone /media/Matrix cifs username=architect,password=$(base64 -d “”),iocharset=utf8,sec=ntlm 0 0

Great! Now I can carry on to my actual goal, which was to nail challenge 91 (“Sysadmin Linux Level 7”)!

RingZer0team CTF - Challenge 57

Fri, 05 May 2017 00:00:00 +1200

This is a continuation of my series on RingZer0Team.com.

Challenge 57 (“Hash Breaker Reloaded”, under the Coding Challenges) is one of a series of challenges where you’re simply presented with a hash - you need to return the plaintext value to the page within 3 seconds.

In contrast to Challenge 56, this challenge also includes a salt:

You have 3 seconds to break this hash
Send the answer back using https://ringzer0team.com/challenges/57/[clear_text]


----- BEGIN HASH -----
ab9507edbb2501b3c02e47c51af0178d68655980
----- END HASH -----

----- BEGIN SALT -----
c2ac9d8d004b4011d0864e76c7ebaaccfd18464bb8ff66bdbf19a703eb95a944
----- END SALT -----

The hash looks like another SHA-1, but of course the online hash reversers don’t have the now-salted hash in their rainbow tables. I figured that the challenge was a simple continuation of the previous challenge, so I crossed my fingers and guessed the hash was simply a number (from last time), concatenated with the plain-text salt.

… And I was lucky! The below code got me the flag:

#!/usr/bin/python3
from requests import get
from hashlib import sha1
from re import search

url='https://ringzer0team.com/challenges/57'
session=dict(PHPSESSID='<blahblah>')
resp=get(url,cookies=session)
hash=resp.text.split("-----")[2].split("\t")[2].split("<")[0]
salt=resp.text.split("-----")[6].split("\t")[2].split("<")[0]
newhash=''
salt=salt.encode('utf-8')
x=0
while newhash != hash:
		newhash=sha1(str(x).encode('utf-8')+salt).hexdigest()
		print("%s = %s" % (newhash, x))
		x+=1
	
resp=get(url+'/'+str(x-1),cookies=session)
flag=search("FLAG-.{24}",resp.text).group()
print(flag)

As last time, there’s no error handling in that code, so if it runs for more than a few seconds it’s probably toast.

RingZer0team CTF - Challenge 56

Tue, 02 May 2017 00:00:00 +1200

Challenge 56 (“Hash Breaker”, under the Coding Challenges) is one of a series of challenges where you’re simply presented with a hash - you need to return the plaintext value to the page within 3 seconds.

Of course, this is impossible to do manually, so it’s a programming challenge.

The hash looks like a simple SHA-1, so I put it into an online hash reverser and discovered it’s just a SHA1 hash of a random number. SHA1 is really fast, so this should be simple.

Here’s how I tackled the problem:

#!python
# It's in Python3, because of reasons
from requests import get
from hashlib import sha1
from re import search
calculated_hash=''
x=0

# Your session cookie needs to go here
session=dict(PHPSESSID='<cookie contents>')

# Load the web page
url='https://ringzer0team.com/challenges/56'
resp=get(url,cookies=session)

# Extract the hash from the page contents
target_hash=resp.text.split("-----")[2].split("\t")[2].split("<")[0]

# Iterate hashes from zero until the calculated hash matches the target hash
while calculated_hash != target_hash:
		calculated_hash=sha1(str(x).encode('utf-8')).hexdigest()
		print("%s = %s" % (calculated_hash, x))
		x+=1

# If we got here, yay! We found the key. Now submit it back to the site, and print our magic flag
resp=get(url+'/'+str(x-1),cookies=session)
flag=search("FLAG-.{24}",resp.text).group()
print(flag)

There’s no error handling in that code, so if it runs for more than a few seconds it’s probably toast.

HP Procurve's warn-and-disable

Sun, 30 Apr 2017 00:00:00 +1200

Since mid-2016, I’ve been working as a Network and Security Administrator. While I’d done a fair amount of networking previously, most of my experience had been with either unmanaged switches, or in a pre-built Cisco environment. Stepping into the world of managed networking was new for me, as was stepping into the world of HP Switches.

We were having recurring issues with a certain business unit looping ports on a switch. We had loop-protect running, but it was only set to disable the port after 5 seconds, and only for 300 seconds. This wasn’t long enough (we’ve since set it to something more resilient).

During our attempts to minimise the impact of a network loop, we enabled this setting on our HP Procurve switches:

fault-finder broadcast-storm action warn-and-disable

This seems simple enough. The fault-finder module will look for broadcast storms, and when found, will throw a syslog, an SNMP trap, and will disable the offending switch port.

Don’t be fooled. This will absolutely wreck your breakfast.

The fault-finder module is a system-wide setting, which means it applies to uplink ports as well. When a broadcast storm happens, all your distribution switches will see the broadcasts coming from… their uplink port. So of course, the switches disable their uplink to the rest of the network, neatly segregating themselves away from any services or central management.

Suffice to say, the next time we had a network loop there was a lot of walking and console cable madness, while we visited each of the distribution switches and manually re-enabled the disabled ports.

Lesson learned, loop-protect it is. (Until we get STP up campus-wide, of course)

Making Lektor work with grsecurity

Sat, 29 Oct 2016 00:00:00 +1300

I started using grsecurity on my servers in 2015, and there’s always a bit of tuning required.

I was recently playing with Lektor (before I swapped to Pelican), and I had a bit of trouble with my grsec kernel. In particular, Lektor and Pelican are both run within a virtualenv Python environment, and grsec eats it like popcorn in two different ways:

TPE (Trusted Path Execution) throws a wobbly:

[253241.370019] grsec: From {ssh-origin-ip}: denied untrusted exec (due to file in world-writable directory) of /tmp/#50 by /usr/local/lib/lektor/bin/lektor[lektor:60593] uid/euid:1000/1000 gid/egid:1000/1000, parent /bin/bash[bash:60581] uid/euid:1000/1000 gid/egid:1000/1000

This block occurs because the virtualenv violates grsecurity’s Trusted Path Execution protection - the Python executable isn’t under a trusted path, and the user isn’t in the TPE-bypass group.

To resolve this one, add your user to the TPE group. This will allow the user to execute binaries which aren’t in trusted locations:

sudo usermod -aG grsec-tpe $USER

RWX protection. This is a common problem with Python-based apps, even though Python has an exception (Edit 2020-04-12 - Issue 6 from https://github.com/thestinger/paxd/ has been deleted) by default.

[ 207.534876] grsec: From {ssh-source-ip}: denied RWX mmap of <anonymous mapping> by /usr/local/lib/lektor/bin/lektor[lektor:534] uid/euid:1000/1000 gid/egid:1000/1000, parent /bin/bash[bash:477] uid/euid:1000/1000 gid/egid:1000/1000

First, create your PaX ELF headers, and then disable grsec’s MEMPROTECT extension on the included python2 binary, while you enable EMUTRAMP:

sudo paxctl /usr/local/lib/lektor/bin/python2 -c
sudo paxctl /usr/local/lib/lektor/bin/python2 -Em

Result!

Broken log times in APC's PCNS Appliance 4.1

Thu, 27 Oct 2016 00:00:00 +1300

Schneider Electric’s PowerChute Network Shutdown is a piece of software which communicates with your local UPS, and initiates system shutdown if the UPS battery is unable to continue providing power. This helps to preserve file integrity in the event of a prolonged power failure.

Previously, you had to install the PCNS client separately on each virtual machine. Since then though, APC have released a PCNS VMware Appliance which is installed directly into vCenter, and initiates shutdown on all the VMware guests through a single Virtual Machine. This is a much tidier model, so we’ve recently embarked on migrating to the PCNS Appliance.

After installing the PCNS 4.1 appliance and getting everything working, I discovered that the time on all the log files was wrong by a significant margin. I can’t remember exactly what the margin was, it could have been say 8 hours or something.

I did a bit of research and eventually I discovered a spectaularly old question on the APC forums, dating all the way back to 2011:

“I have installed PCNS 2.2.4 into the vMA of an ESXi 4.1 server. It properly communicates with the associated NMC and performs correctly. However, it reports all the activities in Eastern Standard Time. I don’t know where it’s picking up that time zone and can’t find anywhere to change it. In the vSphere client the time shows up correctly. Checking date inside the vMA from the CLI also shows the correct time and timezone. The date and time setting in the NMC is correct and is set to synch with NTP with correct time zone.”
From: http://forums.apc.com/spaces/7/ups-management-devices-powerchute-software/forums/general/6809/pcns-2-2-4-esxi-annoyance

The answers include a broken link to an APC web page about it, but fortunately the poster copied the original post detail, which showed me where to look:

“PCNS uses a standard java function to retrieve the current date from the system when writing events to the log. The Problem on the VIMA is that this function always returns the US date format (PDT).
This is an issue with Java http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6456628
You can correct the time issue on VIMA by modifying /etc/sysconfig/clock file.
+/etc/sysconfig/clock contains a line ZONE=“America/Los_Angeles”+
You need to edit the ZONE= to match your region such as ZONE=“America/New_York”

Sounds great! But when I looked in /etc/sysconfig/clock, I discovered my time zone was already set to “Pacific/Auckland”.

However, it also contained the value “UTC = False”. I changed this to “UTC = True”, and et voilà! The problem is fixed!

Updating the modem and radio firmware on a Samsung Galaxy 5 (G900I)

Thu, 27 Oct 2016 00:00:00 +1300

A couple months ago, Check Point revealed their discovery of the Quadrooter vulnerability affecting the Qualcomm chipsets in oodles of Android phones.

I use –CyanogenMod– LineageOS on a Samsung Galaxy S5, so thankfully I received patches for three of the four vulnerabilities in only a few days.

However, that last vulnerability was part of a proprietary binary blob for controlling the Qualcomm LTE chipset, only patchable by Samsung themselves. It took a while for the Samsung updates to roll out, and then I got distracted for a while, but I finally downloaded my firmware. The package I’ve used is “G900IDVU1CPH3”, which seems to be compatible at least with the three NZ-based carriers, as well as all the Australian carriers.

There are a couple little issues with the firmware upgrade, easily fixed if you know how. I’m going to assume you know exactly how to do everything without help. I’m sure you’re pleased.

In particular, the version of Heimdall in the Debian Sid repositories (1.4.1-2) doesn’t quite work with the SGS5. It appears other people have had similar problems.

The beginning of fix for me came from turboyz on Github, at the bottom of this post: (Edit 2020-04-12: GitHub Issue 348 from https://github.com/Benjamin-Dobell/Heimdall/ has been deleted); however, he’s manually made a couple changes to BridgeManager.cpp, which appear to be no longer necessary. We can simply build it from source:

sudo apt-get install build-essential cmake zlib1g-dev libusb-1.0-0-dev git
git clone https://github.com/Benjamin-Dobell/Heimdall.git
cd Heimdall

# OPTIONAL: Remove the `if(NOT DISABLE_FRONTEND)` codeblock at the end of Heimdall/CMakeLists.txt
# I believe this just saves time compiling the GUI

mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make
cd bin

Now you’ve got a nice new Heimdall installation, with the approproate updates to allow it to talk nicely to the new Samsung bootloader.

I copied my new baseband files into the Heimdall/build/bin folder, just to get them all together. The last step is to flash your NON-HLOS.bin and modem.bin files:

./heimdall flash --APNHLOS NON-HLOS.bin --MODEM modem.bin

I happen to have an issue on my phone currently where, even though the modem and LTE drivers have both been successfully installed, and even thought the Quadrooter tester is no longer showing any vulnerabilities, my Baseband version on my phone is still shown as the old one. I haven’t been able to figure out why yet, but I don’t think I’ll worry about it for now. There have been reports that you should first do this with –no-reboot, wait for the update to complete, then pull your battery, then when you power back on, immediately go back into download mode and re-flash. Apparently this helps update the recognised Baseband version, but to be honest it didn’t help me.

If you’re completely 100% lucky and absolutely nothing goes wrong, you win! You should now have the most recent Samsung modem and LTE chipset drivers available for your phone… without having to roll back to the stock Samsung firmware.

Incidentally, if you want to save gigabytes and time downloading said firmware, and if your phone is precisely a Samsung Galaxy S5 (G900I), and if you definitely want firmware G900IDVU1CPH3, I’ve extracted the important files (modem.bin and NON-HLOS.bin) and stored them here:

https://www.ericlight.com/files/G900IDVU1CPH3_modem_LTE.zip
SHA256 hash = a2ab13063583f6e83a3c2d8b79521a59ba103dda30a24bddf9248dd5a25bff3c

I promise I haven’t intentionally backdoored them, but I make no warranties of any sort. They might just be pictures of my foot.

Good luck!

Using ECDSA certificates with Let's Encrypt

Thu, 27 Oct 2016 00:00:00 +1300

Let’s Encrypt’s Certbot will generate an RSA key by default. But we want to step into the new and exciting world of elliptic curve cryptography! Unfortunately Certbot doesn’t really roll that way, so there are a couple hoops to jump through first.

A word of caution: this post has been written in retrospect, some time after I actually got ECDSA working. That means there are bound to be squiggly little steps that I’ve missed, and I certainly should have provided screenshots or snippets that I’ve missed. Sorry.

This post assumes you’ve already installed Certbot. I had a working regular certificate from Certbot before I changed to ECDSA, so if you have problems following this from scratch, I do recommend trying that first.

For future reference, I’m running the latest certbot available in Debian unstable, which is version 0.8.1-3.

I got most of this information from Scott Helme’s website, which has been awesome.

In brief:

Generate yourself an ECDSA private key:

openssl ecparam -genkey -name secp384r1 | openssl ec -out ec.key

You can change the curve that you use, if you feel a bit wiggly about the controversy around the NSA & NIST degrading the quality of the curves. I don’t feel particularly wiggly about that, myself.

Generate a Certificate Signing Request (CSR) with your shiny new key:

openssl req -new -sha256 -key ec.key -nodes -out ec.csr -outform pem

That will give you ec.key (your private key), and ec.csr (your certificate signing request). Time to get Let’s Encrypt involved.

Create your certificate:

certbot certonly -w /var/www/html/ -d {your_domain} –email “{your_email}” –csr ./ec.csr –agree-tos

If everything goes perfectly, that should leave you with a new shiny set of certificates – quite possibly named something clumsy like 0000-cert.pem and 0001-fullchain.pem, or similar. Throw those into your nginx config and give it a test to see if it’s working.

Schedule your certificate renewals:

I had particular trouble with the renewal process of ECDSA certificates, because certbot renew isn’t compatible with custom CSR’s. You need to run certbot certonly to pass the –csr argument, and then you need to deal with the output yourself.

Even more irksome, the certonly function will fail if you ask it to renew certificates which already exist:

An unexpected error occurred:
OSError: [Errno 17] File exists: '/etc/letsencrypt/live/{your_domain}/cert_ecdsa.pem'
Please see the logfiles in /var/log/letsencrypt for more details.

There doesn’t seem to be any way to tell certbot to overwrite the old certificates automatically, so I created a /etc/letsencrypt/temp folder, and wrote up a really yuck cron job for it. If I were working on a production system I’d do something better, but this works for my lowly domain:

# Recreate certs under /etc/letsencrypt/temp
30 2 24 * * certbot certonly -w /var/www/html/ -d {your_domain} --email "{your_email}" --csr /path/to/your/ec.csr --agree-tos --non-interactive --webroot --cert-path /etc/letsencrypt/temp/cert_ecdsa.pem --fullchain-path /etc/letsencrypt/temp/fullchain_ecdsa.pem

# Backup current certs to home dir
31 2 24 * * mv /etc/letsencrypt/live/{your_domain}/cert_ecdsa.pem ~/certbackup/
31 2 24 * * mv /etc/letsencrypt/live/{your_domain}/fullchain_ecdsa.pem ~/certbackup/

# Move new certs to live folder
32 2 24 * * mv /etc/letsencrypt/temp/* /etc/letsencrypt/live/{your_domain}/

# Restart nginx
33 2 24 * * service nginx restart

And that’s all! It seems to be working so far, but I’m sure something is going to fail at some point – maybe I’ll hit my request limit, or the Let’s Encrypt service will be down, and I’ll lose my certs. If it happens too often I’ll come up with a nicer cronjob and update this post.

Tidying up Pelican's URLs with the Flex theme

Sun, 23 Oct 2016 00:00:00 +1300

This is just a brain-dump of tweaks I’ve made to my Pelican environment to get tidy URLs. In summary: One theme tweak, one Nginx tweak.

Making the Flex Theme drop the ugly url#slug format:

I’ve decided I’m going to roll with the Flex theme for a while. I like it, but it had this habit of putting anchors and slugs into my article URLs.

For example, my first page is known to Pelican as “welcome-to-the-internet”. However, links to the page were automatically created as “welcome-to-the-internet#welcome-to-the-internet”. It seems redundant. Also, it says the same thing twice. It repeats itself.

I discovered the issue in the Flex theme. It’s under /templates/index.html, circa lines 7 and 29. Simply change:

{{ article.url }}#{{ article.slug }}

… to…

{{ article.slug }}

Fixed!

I haven’t submitted a pull request to Alexandre, because I suspect it’s intentional.

Getting Nginx to serve tidy urls:

By default, Pelican creates great urls like {server}/Ramblings/welcome-to-the-internet

This is great, but it CREATES files like /var/www/Ramblings/welcome-to-the-internet.html

Nginx by default doesn’t see “welcome-to-the-internet” and automatically serve “welcome-to-the-internet.html”. The lack of a file extention means that the .html file is fundamentally a different file, so it returns a handy-dandy 404 error instead.

To fix this, open /etc/nginx/sites-available/yoursite, and find:

location / {
 try_files $uri $uri/ =404;
}

Change it to the following - note I’ve added $uri.htm and $uri.html, to inform Nginx that it should try appending .htm or .html onto the end of a uri if it can’t find the page:

location / {
 try_files $uri.htm $uri.html $uri =404;
}

All done, happy tidy URLs!

Alfredo Sauce

Sat, 22 Oct 2016 00:00:00 +1300

I had the urge to make a homemade alfredo the other night. I found a recipe called “Better than Olive Garden Alfredo Sauce”, which is pretty descriptive, I guess. I made a couple tweaks here and there, though.

I usually stray wildly from recipies whenever I decide to follow one, but having not made an alfredo sauce before, I largely stuck to the guide. A couple changes I did make, though:

Instead of adding the garlic and cream together, I heated the butter and fried the garlic to bring out a bit more flavour
I also fried a goodly dose of smoked bacon along with the garlic. Bacon first, of course, so the garlic doesn’t burn
The recipe reckoned we should use two garlic cloves. Ha! Four, definitely.
I also added some chopped spring onion and a pinch of cayenne pepper

I’ve reproduced the origina recipe, along with my changes, here:

Ingredients

3 tablespoons butter
2 tablespoons olive oil
2 or 3 rashers of bacon, chopped into squares
4 garlic cloves, minced
1 chopped spring onion
2 cups cream
1⁄4 teaspoon freshly-ground pepper
1 pinch cayenne pepper
1⁄2 cup grated parmesan cheese
3⁄4 cup mozzarella cheese
1 1/2 supermarket tubes of fettucini pasta

Method

Melt butter in medium saucepan with olive oil over medium/low heat.
Fry the bacon until until browned.
Add the garlic, fry until the garlic starts to brown.
Add cream and spring onion; bring mixture to a simmer.
Stir often.
Add the Parmesan cheese and simmer sauce for 8-10 minutes or until sauce has thickened and is smooth.
When sauce has thickened add the Mozzarella cheese and stir until smooth. STIR FREQUENTLY.
Add the pepper and cayenne.
While the sauce cooks, boil noodles for 3-5 minutes.
Place pasta on serving plates and spoon sauce over pasta.

Bone app the teeth!

(Alright, most of the reason for this blog post is for me to practice my Markdown skills)

Welcome to the Internet

Sat, 15 Oct 2016 00:00:00 +1300

Well, I’m finally here.

I’ve never really run a blog before… I did write a bunch of stuff back when I ran Gravity Computing; it just was never really “bloggy”. That content has now been moved to Efficient Software, but I’ll probably copy my parts back over here when time permits.

I really have no intention of running a blog, but I do come across interesting and challenging problems all the time, which I’d like to share with people who are, apparently, just like you.

I’ve got a couple things saved up that I want to post about in the next few days, but heck this site could languish for months at a time – so feel free to add it to your reader, I definitely won’t fill your feed.