Tech on Such geek. Wow.

task pve-bridge blocked for more than 120 seconds - on Proxmox with MDATP

Sun, 13 Mar 2022 00:00:00 +1300

Today I updated and rebooted a bunch of servers - all in a day’s work. Everything went fine, until one of the hosts (the big one) didn’t boot up again properly.

I’ve got two hosts, both running ProxmoxVE (version 7.1) and Microsoft Defender for Endpoint (version 101.61.69-insiderfast). On reboot, one of the hosts came up fine; the other didn’t.

Long story short, uninstalling mdatp fixed the problem - wild hypothesis, but I suspect it was interfering with either network bridge or interprocess communication.

The rest of this post is technical info for people who might be able to fix it, and so that this blog post will come up for the other people who inevitably bump into this!

Symptoms - upon boot, the host came up, was apparently able to launch one VM, and recieved a timeout for all subsequent VM’s:

The lines are truncated, but all those pink lines are showing TASK ERROR: start failed: command '/usr/bin/kvm -id 153 [...]' failed: got timeout.

I say ‘apparently’ launched one VM, because I was unable to interact with it - in fact all the launch-on-boot VM’s appeared to have booted, except everything about them was non-responsive, including Proxmox stats windows and console.

Trying to manually interact with the VM’s using qm failed entirely - the console froze, completely unresponsive to either ^C or ^X, until I forcefully disconnected from SSH.

Checking dmesg revealed a horror story:

Mar 13 16:54:36 buckwheat kernel: [ 159.853700] vmbr1: port 1(eth3) entered blocking state
Mar 13 16:54:36 buckwheat kernel: [ 159.853703] vmbr1: port 1(eth3) entered forwarding state
Mar 13 16:54:43 buckwheat kernel: [ 166.700586] ixgbe 0000:81:00.1 eth3: NIC Link is Down
Mar 13 16:54:43 buckwheat kernel: [ 166.701135] vmbr1: port 1(eth3) entered disabled state
Mar 13 16:54:45 buckwheat kernel: [ 168.294208] ixgbe 0000:81:00.1 eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX
Mar 13 16:54:45 buckwheat kernel: [ 168.294343] vmbr1: port 1(eth3) entered blocking state
Mar 13 16:54:45 buckwheat kernel: [ 168.294349] vmbr1: port 1(eth3) entered forwarding state
Mar 13 16:54:47 buckwheat pvestatd[5040]: VM 112 qmp command failed - VM 112 qmp command 'query-proxmox-support' failed - unable to connect to VM 112 qmp socket - timeout a>
Mar 13 16:54:50 buckwheat pvestatd[5040]: VM 105 qmp command failed - VM 105 qmp command 'query-proxmox-support' failed - unable to connect to VM 105 qmp socket - timeout a>
Mar 13 16:54:53 buckwheat pvestatd[5040]: VM 115 qmp command failed - VM 115 qmp command 'query-proxmox-support' failed - unable to connect to VM 115 qmp socket - timeout a>
Mar 13 16:54:55 buckwheat pve-guests[11257]: start failed: command '/usr/bin/kvm -id 153 -name Eden -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/153.qmp,s>
Mar 13 16:54:56 buckwheat pvesh[5530]: Starting VM 153 failed: start failed: command '/usr/bin/kvm -id 153 -name Eden -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qem>
Mar 13 16:54:56 buckwheat pve-guests[5530]: <root@pam> end task UPID:buckwheat:000015C7:00000710:622D6A6F:startall::root@pam: OK
Mar 13 16:54:56 buckwheat systemd[1]: Finished PVE guests.
Mar 13 16:54:56 buckwheat systemd[1]: Reached target Multi-User System.
Mar 13 16:54:56 buckwheat systemd[1]: Reached target Graphical Interface.
Mar 13 16:54:56 buckwheat systemd[1]: Starting Update UTMP about System Runlevel Changes...
Mar 13 16:54:56 buckwheat systemd[1]: systemd-update-utmp-runlevel.service: Succeeded.
Mar 13 16:54:56 buckwheat systemd[1]: Finished Update UTMP about System Runlevel Changes.
Mar 13 16:54:56 buckwheat systemd[1]: Startup finished in 4.603s (kernel) + 2min 54.823s (userspace) = 2min 59.427s.
Mar 13 16:54:56 buckwheat pvestatd[5040]: VM 104 qmp command failed - VM 104 qmp command 'query-proxmox-support' failed - unable to connect to VM 104 qmp socket - timeout a>
Mar 13 16:54:59 buckwheat pvestatd[5040]: VM 153 qmp command failed - VM 153 qmp command 'query-proxmox-support' failed - got timeout
Mar 13 16:54:59 buckwheat pvestatd[5040]: status update time (30.177 seconds)
Mar 13 16:55:18 buckwheat pvestatd[5040]: VM 104 qmp command failed - VM 104 qmp command 'query-proxmox-support' failed - unable to connect to VM 104 qmp socket - timeout a>
Mar 13 16:55:21 buckwheat pvestatd[5040]: VM 153 qmp command failed - VM 153 qmp command 'query-proxmox-support' failed - unable to connect to VM 153 qmp socket - timeout a>
Mar 13 16:55:24 buckwheat pvestatd[5040]: VM 112 qmp command failed - VM 112 qmp command 'query-proxmox-support' failed - unable to connect to VM 112 qmp socket - timeout a>
Mar 13 16:55:27 buckwheat pvestatd[5040]: VM 105 qmp command failed - VM 105 qmp command 'query-proxmox-support' failed - unable to connect to VM 105 qmp socket - timeout a>
Mar 13 16:55:30 buckwheat pvestatd[5040]: VM 115 qmp command failed - VM 115 qmp command 'query-proxmox-support' failed - unable to connect to VM 115 qmp socket - timeout a>
Mar 13 16:55:30 buckwheat pvestatd[5040]: status update time (30.186 seconds)
Mar 13 16:55:48 buckwheat pvestatd[5040]: VM 153 qmp command failed - VM 153 qmp command 'query-proxmox-support' failed - unable to connect to VM 153 qmp socket - timeout a>
Mar 13 16:55:51 buckwheat pvestatd[5040]: VM 104 qmp command failed - VM 104 qmp command 'query-proxmox-support' failed - unable to connect to VM 104 qmp socket - timeout a>
Mar 13 16:55:54 buckwheat pvestatd[5040]: VM 115 qmp command failed - VM 115 qmp command 'query-proxmox-support' failed - unable to connect to VM 115 qmp socket - timeout a>
Mar 13 16:55:57 buckwheat pvestatd[5040]: VM 105 qmp command failed - VM 105 qmp command 'query-proxmox-support' failed - unable to connect to VM 105 qmp socket - timeout a>
Mar 13 16:56:00 buckwheat pvestatd[5040]: VM 112 qmp command failed - VM 112 qmp command 'query-proxmox-support' failed - unable to connect to VM 112 qmp socket - timeout a>
Mar 13 16:56:00 buckwheat pvestatd[5040]: status update time (30.191 seconds)
Mar 13 16:56:00 buckwheat kernel: [ 243.301563] INFO: task wdavdaemon:5583 blocked for more than 120 seconds.

The wdavdaemon is shown blocking here, but pve-bridge turned up just as frequently in the logs:

root@buckwheat:/var/log# grep blocked syslog
Mar 13 18:34:14 buckwheat kernel: [ 242.802463] INFO: task wdavdaemon:5740 blocked for more than 120 seconds.
Mar 13 18:34:14 buckwheat kernel: [ 242.802658] INFO: task wdavdaemon:4899 blocked for more than 120 seconds.
Mar 13 18:34:14 buckwheat kernel: [ 242.802840] INFO: task pve-bridge:6117 blocked for more than 120 seconds.
Mar 13 18:34:14 buckwheat kernel: [ 242.802945] INFO: task pve-bridge:7734 blocked for more than 120 seconds.
Mar 13 18:34:14 buckwheat kernel: [ 242.803057] INFO: task pve-bridge:9120 blocked for more than 120 seconds.
Mar 13 18:34:14 buckwheat kernel: [ 242.803170] INFO: task pve-bridge:10283 blocked for more than 120 seconds.
Mar 13 18:36:15 buckwheat kernel: [ 363.631430] INFO: task wdavdaemon:5740 blocked for more than 241 seconds.
Mar 13 18:36:15 buckwheat kernel: [ 363.631610] INFO: task wdavdaemon:4899 blocked for more than 241 seconds.
Mar 13 18:36:15 buckwheat kernel: [ 363.631776] INFO: task pve-bridge:6117 blocked for more than 241 seconds.
Mar 13 18:36:15 buckwheat kernel: [ 363.631872] INFO: task pve-bridge:7734 blocked for more than 241 seconds.

So the interesting thing is that another node - foxtail - was also updated to the same version of both Proxmox and mdatp. The only obvious difference I can see are the numbers of VLANs each of them are bridging to:

This is a list of the bridges now, when they’re both working (sorry, Markdown hates tabs; the below should be four tidy columns):

root@foxtail:/var/log# brctl show
bridge name	bridge id		STP enabled	interfaces
fwbr101i0	8000.9e7164f22bab	no		fwln101i0
							tap101i0
vmbr0		8000.c2602668d5f4	no		eno1
							fwpr101p0
							tap100i0
							tap109i0

versus

root@buckwheat:/var/log# brctl show
bridge name	bridge id		STP enabled	interfaces
fwbr102i0	8000.6	abd66965438	no		fwln102i0
							tap102i0
fwbr104i0	8000.f60b3f1a3118	no		fwln104i0
							tap104i0
fwbr105i0	8000.961281d6d5dc	no		fwln105i0
							tap105i0
vmbr0		8000.90e2bad572a8	no		eth2
							tap112i0
							tap115i0
							tap153i0
vmbr0v5		8000.0ea35ee013ba	no		eth2.5
							fwpr104p0
vmbr0v666	8000.1a23193ccf17	no		eth2.666
							fwpr102p0
							fwpr105p0
vmbr1		8000.90e2bad572a9	no		eth3

But when buckwheat was broken, the majority of the bridges were not visible:

At some point during troubleshooting, I restarted pvedaemon - although the service had started correctly, the restart failed with error: timeout waiting on systemd.

Acronis Cyber Protect and Microsoft Defender on the same Linux system

Sun, 07 Nov 2021 00:00:00 +1300

A few months ago, I published what I learned from playing with the Linux version of Microsoft Defender for Endpoint a few months back. If you’re here, you’ll have possibly already read my original post. There was a bit of tuning required to get it to behave nicely; just adding some sensible audit logging exclusions.

Well the Acronis Cyber Protect backup platform has recently been added to the bundle, and it had a few specific requirements, so I’ve documented them here.

There are four Acronis processes which MDATP will spend a lot of time scanning and rescanning. Exclude those from the scanning engine by executing the following commands as root:

mdatp exclusion process add --path /usr/lib/Acronis/Schedule/schedul2-bin
mdatp exclusion process add --path /usr/lib/Acronis/APL/active-protection
mdatp exclusion process add --path /opt/acronis/aakore
mdatp exclusion process add --path /usr/lib/Acronis/BackupAndRecovery/mms

There are also five processes which need to be excluded from auditctl, unless you want your audit logs rotated every ten minutes. Again, these must all be run as root.

echo -a never,exit -S 41 -S 42 -F comm="mms" -k exclude_acronis >> /etc/audit/rules.d/01-exclusion.rules
echo -a never,exit -S 41 -S 42 -S 43 -F comm="adp-agent" -k exclude_acronis >> /etc/audit/rules.d/01-exclusion.rules
echo -a never,exit -S 41 -S 42 -S 43 -S 288 -F exe=/usr/lib/Acronis/APL/active-protection -k exclude_acronis >> /etc/audit/rules.d/01-exclusion.rules
echo -a never,exit -S 41 -S 42 -S 43 -F exe=/opt/acronis/bin/updater -k exclude_acronis >> /etc/audit/rules.d/01-exclusion.rules
echo -a never,exit -S 41 -S 42 -S 288 -F exe=/opt/acronis/aakore -k exclude_acronis >> /etc/audit/rules.d/01-exclusion.rules

Once you’ve created the exclusions, you can either reboot or run service auditd restart. Either way, make sure you run service auditd status to ensure nothing went wrong with the exclusions.

Microsoft Defender for Endpoint (mdatp) on Debian Sid

Wed, 07 Jul 2021 00:00:00 +1200

** 2021-08-01 Update: **

Linux doesn’t have many great antivirus options available.

Don’t get me wrong, there are actually a few options nowadays. We’ve got the ever-present ClamAV; BitDefender has a good reputation, but I haven’t played with it yet; Sophos AV for Linux seems to be in limbo. ESET for Linux exists, as does Symantec Endpoint Protection for Linux. That’s a pretty decent range - but they each have some real drawback. Assuming the price is right, you still see compatibility, usability, licensing dramas, or resource drain challenges.

Most recently, the Linux version of Microsoft Defender for Endpoint entered the playing field. I finally got to play with this and I gotta say…

Holy shit.

Look, I know we enjoy ragging on Microsoft. They’ve demonstrated some awful behaviour over the years, and every action still harkens back to the “Embrace, Extend, Extinguish” modus operandi. But they’re doing really great work with the Defender/DATP/MDFE/whatever line. Even the basic Defender has been steading climbing the ranks, to the point where virtually all guidance I’ve heard for the last two years is: “if you’re not going full EDR, just use regulr Defender”. It’s no SentinelOne or CrowdStrike, but overall Defender just Gets The Job Done.

So, I was excited to try the Linux version. Long story short, it was a dream. The installation process was streaks ahead of the competition. Resource usage is generally negligible, adding an average load of around 0.5% CPU, going up to about 16% CPU during a scan. There’s a robust health and connectivity test built-in. Basically the whole thing just works, to the same level we’ve learned to expect from the Windows version. Oh and this one has actual documentation which seems to be considered optional by some of the competitors. 🙃

However.

You may find that mdatp is frantically scanning files that freqently Do Things. Bind9 (named) for example, might be handling hundreds of queries a minute. Each one of them triggers not only an mdatp scan, but also auditd entries… which often logs more than once per query. I was finding logs being filled and rotated every seven minutes in some cases… and then punctuated by auditd yelling “my buffer is full, I can’t log all these events!”.

So as a result, I’ve spent quite a few hours learning how to wrangle auditd around exclusions. Here’s what I’ve learned:

You need to lie to install MDFE on Sid

Edit /etc/os-release and change VERSION_ID="11" to VERSION_ID="10". After that, you can install MDFE following the regular instructions. Don’t forget to change it back later!

Troubleshooting performance issues:

First step - find the processes that mdatp is spending the most CPU time on:

# Download Microsoft's parser (only do this once)
wget -c https://raw.githubusercontent.com/microsoft/mdatp-xplat/master/linux/diagnostic/high_cpu_parser.py

# Ask Defender to output the stats, and then parse it looking for high CPU:
$ mdatp diagnostic real-time-protection-statistics --output json | python high_cpu_parser.py
925 	mongodb		29184
37575	pulseaudio	47
21355	packagekitd	30
37588	dbus-daemon	26
37550	sshd		25

… Yikes, it’s probably not necessary to scan MongoDB thirty thousand times! You can exclude a process from mdatp like this: sudo mdatp exclusion process add --path /usr/bin/mongod

But that’s not all… even though mdatp is no longer scanning the application, you still have auditd frantically logging every single thing it does. That’s also going to cause periodic slowdowns as the auditd buffer fills up, overflows, panicks, and then purges the stuff it couldn’t write down.

Syslog/audit logging issues

You can find out if anything is spewing a billion entries into your audit log, by just grepping for the rotation in syslog:

# grep 'Audit daemon rotating' /var/log/syslog
Jul 6 01:00:08 spam auditd[28996]: Audit daemon rotating log files
Jul 6 04:00:05 spam auditd[28996]: Audit daemon rotating log files
Jul 6 06:57:01 spam auditd[28996]: Audit daemon rotating log files
Jul 6 09:57:18 spam auditd[28996]: Audit daemon rotating log files

… Filling up a log file every three hours is weird, but previously this server was doing it every few minutes. You can see what’s causing all the log rotations with:

# cat /var/log/audit/audit.log* | cut -d ' ' -f26 | sort | uniq -c
 138581 
 1 exe="/bin/bash"
 4 exe="/bin/hostname"
 2 exe="/bin/nano"
 684 exe="/lib/systemd/systemd"
 4754 exe="/lib/systemd/systemd-logind"
 209 exe="/lib/systemd/systemd-user-runtime-dir"
 119 exe="/opt/microsoft/mdatp/sbin/osqueryi"
 9 exe="/usr/bin/apt-get"
 2133 exe="/usr/bin/dbus-daemon"
 13212 exe="/usr/bin/perl"
 10784 exe="/usr/bin/pmxcfs"
 14 exe="/usr/bin/rrdcached"
 2 exe="/usr/bin/sort"
 4 exe="/usr/lib/postfix/sbin/cleanup"
 880 exe="/usr/lib/postfix/sbin/pickup"
 175 exe="/usr/lib/postfix/sbin/qmgr"
 27 exe="/usr/lib/postfix/sbin/smtp"
 4 exe="/usr/lib/postfix/sbin/trivial-rewrite"
 16 exe="/usr/sbin/cron"
 10326 exe="/usr/sbin/ebtables-legacy"
 10326 exe="/usr/sbin/ebtables-legacy-restore"
 3 exe="/usr/sbin/postdrop"
 5 exe="/usr/sbin/qmeventd"
 2 exe="/usr/sbin/sendmail"
 56 exe="/usr/sbin/smartd"
 3255 exe="/usr/sbin/sshd"

The number on the left shows the number of times that binary has had it’s activity logged. This should clearly show the problem executables… today we’re looking at perl, pmxcfs, and ebtables-legacy. Once you’ve got that, you need to determine which syscalls are part of regular noise:

# grep /usr/bin/pmxcfs /var/log/audit/audit.log* | cut -d ' ' -f 4 | sort | uniq -c
 10856 
 8142 syscall=263
 1357 syscall=43
 1357 syscall=84

Referring to this document, we can see these syscalls are sys_accept, sys_rmdir, and sys_unlinkat. Since pmxcfs is the ProxmoxVS Cluster File System, these are frankly all pretty normal syscalls for it to make. Let’s ignore them.

echo -a never,exit -S 43 -S 84 -S 263 -F exe=/usr/bin/pmxcfs -k exclude_PVE_internals >> /etc/audit/rules.d/01-exclusion.rules

^ Gotta do that as root, btw, or just append the line however you see fit. Once you’ve added exclusions for all of your high-noise entires, run service auditd restart and then service auditd status to make sure it worked properly.

On my Zentyal box I had to add a bunch of entries to exclude logging of certain high-noise things. For example, DNS query resolution (via named) and Samba activity:

cat /etc/audit/rules.d/01-exclusion.rules

-a never,exit -S 41 -S 42 -S 49 -S 82 -S 288 -F exe=/usr/sbin/named -k exclude_DNS_queries
-a never,exit -S 43 -F exe=/usr/sbin/winbindd -k exclude_Samba
-a never,exit -S 41 -S 42 -S 43 -S 87 -F exe=/usr/sbin/smbd -k exclude_Samba
-a never,exit -S 41 -S 43 -S 87 -F exe=/usr/sbin/samba -k exclude_Samba

A word of warning: Don’t just ignore everything willy-nilly. The audit logging system exists so you can track activity on the system - blithely sending 100% of the log entries to the bin will undermine what the audit sytem exists for!

… That’s all I’ve got so far. I’ll keep updating this if I bump into anything new!

Configuring a custom SSL certificate in Zentyal

Sun, 25 Apr 2021 00:00:00 +1200

Zuper-quick post for when this bites me again in the future!

Tonight I upgraded from Zentyal 6.2 to Zentyal 7.0. Smooth as butter, everything went great.

Until I logged in, and my shiny Actual Paid Money SSL certificate had vanished, replaced by a self-signed commoner’s certificate. Piffle.

Instructions for installing a custom SSL certificate in Zentyal are actually kinda shaky, so here you go:

cd /var/lib/zentyal/conf/ssl/
mkdir old
mv * old
nano ssl.key
nano ssl.cert
cat ssl.cert ssl.key > ssl.pem
chmod 0400 ssl*

That’s it! Your Private Key lives in ssl.key, and your SSL certificate lives in ssl.cert.

Reboot, and job done.

Blocking bad extensions and extortion with iRedMail

Mon, 12 Apr 2021 00:00:00 +1200

Turns out this is my third Amavis article. I guess it’s just one of those systems.

Heads up, if you’re trying to do this, make sure you also read my other article about the “banned_files_lover” thing.

Today I’m on a mission to:

Drop all incoming Office ‘97 files (they’re predominantly malicious these days)
Drop all incoming Macro-enabled Office 2007+ files (there aren’t legitimate reasons to receive these in my scenario)
Drop any emails containing a .onion address
Drop any emails containing a bitcoin wallet

Dropping Attachments

The attachment block is easily handled by Amavis. In iRedMail on Debian, the configuration file is found at /etc/amavis/conf.d/50-user. Open your config file, and scroll down to the section where the $banned_filename_re variable is set. Insert the following line:

 qr'.\.(doc|dot|docm|docb|xls|xlm|xlt|xlsm|xlsb|
 xla|xlam|ppt|pps|pptm|potm|ppam|ppsm|sldm)$'i, # Office '97-2003 and Macro-enabled files
 qr'.\.(adn|accdb|accdr|accdt|accda|mdw|accde|mam|maq|mar|mat|
 maf|laccdb|ade|adp|mdb|cdb|mda|mdn|mdt|mdf|mde|ldb)$'i, # Microsoft Access files
 qr'^\.pub$', # Microsoft Publisher files

That will block all core Office ‘97-2003 files, as well as all Macro-enabled Office 2007-365 files.

By default, this will silently reject mails containing these attachments. If you want senders to receive a bounce message, search for the $final_banned_destiny variable and make sure it’s set to D_BOUNCE.

Dropping .onion and BTC

This part happens in Postfix, and it’s more-super-easy than the last bit. To configure Postfix’s body checks, edit /etc/postfix/body_checks.pcre. I simply added the following:

/(\w+\.onion)/ REJECT This mail server does not accept references to .onion addresses (${1})
/(\b(bc1|[13])[a-zA-HJ-NP-Z0-9]{25,39})/ DISCARD Bitcoin wallet detected (${1})

I used the REJECT directive for testing, but to be honest I don’t want to send bounces back to these people, so I changed to DISCARD after it was working. You can see the difference above.

The text after the REJECT directive is returned to the sender, and the text after the DISCARD directive is logged. The variable ${1} contains the detected string, and is appended to the response message.

That’s the end! Safety. Yes.

Acknowledgements

Thanks to Brad and Hamish for this post - Brad for the regex, Hamish for the idea, and both of them for the review!

Using Caddy to enable MTA-STS

Tue, 19 Jan 2021 00:00:00 +1300

About 7 months ago, I left Nginx and moved to Caddy. I’ve found it super easy, and have now experienced using it as a reverse proxy, a static site server (this one!), as well as a hosting a handy place for me to copy my favourite Kaomoji.

Update 2021-01-20: I had a bug! By default, Caddy serves this without a Content-Type header. However, the RFC specifies that the response should be explicitly Content-Type "text/plain". Thanks to klausenbusk for letting me know!

Note: the Caddyfile fragment to generate https://shrug.ericlight.com looks like this:

shrug.ericlight.com { respond "<meta charset='UTF-8'>¯\_(ツ)_/¯" }

Anyway, once I’d used Caddy’s respond directive, it was a simple step to take it forward and use it to serve my handy-dandy MTA-STS policy file!

Simply add this new Server block to your Caddyfile:

mta-sts.ericlight.com {

respond "version: STSv1
mode: testing
mx: in1-smtp.messagingengine.com
mx: in2-smtp.messagingengine.com
max_age: 86401"

header Content-Type "text/plain; charset=utf-8"

}

This simple Server block will get you a shiny HTTPS certificate, redirect any HTTP visitors to HTTPS, and will provide enquirers with your MTA policy file. (Obviously, replace my domain name and MX entries with your own.)

Sharp observers will notice that the MTA-STS policy file should be hosted at https://mta-sts.ericlight.com/.well-known/mta-sts.txt… but that’s the elegance of the Respond directive in Caddy: wherever you go at mta-sts.ericlight.com, you get exactly that response!

Example: https://mta-sts.ericlight.com/my/hovercraft/is/full/of/eels.txt

Okay, that was easy… the next part is just creating your three DNS records:

Record #1: An A record for mta-sts.ericlight.com, pointing to your Caddy server.
Record #2: A TXT record at _mta-sts.ericlight.com, reading v=STSv1; id=20210112. (Note the underscore! Also, the ID can be anything.)
Record #3: A TXT record at _smtp._tls.ericlight.com, reading v=TLSRPTv1; rua=mailto:tls-reports@targetdomain.com. (update that email address)

… There you go! Now you have a shiny new MTA-STS policy, and your Reporting email target (configured in DNS Record #3) will start reciving daily reports from sending mail servers, letting you know if your policy is working correctly.

Eventually (once you’re sure it’s all configured properly), you should change your testing policy to enforce. When you do this, make sure you update the ID in your TXT Record! This is how sending mail servers know that there has been a policy update.

Enjoy!

Getting TrueNAS (FreeNAS 12) to work with an APC NMC2 UPS (AP9631)

Thu, 14 Jan 2021 00:00:00 +1300

TrueNAS, and FreeNAS before it, has supported UPS via the NUT software package since forever. But most people using it seem to be using USB-connected UPS devices. I don’t have one of these. I’m dealing with an IPv4-based APC ups, specifically the Smart-UPS X 1500 (SMX1500RMI2UNC), with an AP9631 NMC2 card.

You’re probably here because you’re in the same boat: trying to set up a non-USB or Ethernet-based UPS under FreeNAS/TrueNAS, and you can’t. You’ve noticed the TrueNAS console will not shut up about errors which read something like:

nut plugin: nut_connect: upscli_connect (localhost, 3493) failed: Connection failure: Connection refused. (Or perhaps port 161)

“But why?”, you say. “I’ve given TrueNAS my UPS hostname, why is it always trying to talk to localhost? And what is this port 3493, when SNMP is 161?”

What’s happening is that upscli is trying to talk to the upsd damon on your TrueNAS server, which is expected to listen on port 3493 - it’s upsd which then does the talky with the SNMP driver, which in turn does the talky with the UPS. This is why the TrueNAS console logs look like it’s just shouting at itself.

Okay, let’s get into it:

UPS Configuration

First, you need to enable SNMP on your AP9631. This was pretty straightforward, but remember that you need to go:

Configuration -> Network -> SNMPv1 -> Access -> Enable
Configuration -> Netowrk -> SNMPv1 -> Access Control, and make sure the public community is set to be accessible from 0.0.0.0
Reboot your NMC2 card.

This configuration should work with the defaults that the NUT SNMP driver expects - I suggest you get it working like this for now, then tweak to SNMPv3 once you have a confirmed-working connection.

TrueNAS Configuration

Your UPS configuration menu is under TrueNAS -> Services -> UPS. Configure your TrueNAS UPS section like so:

General Options section:

Identifier: Up to you, but do yourself a favour and make it ups.
UPS Mode: Master
Driver: “Various ups 3 (various) SNMP - RFC 1628 (snmp-ups, experimental)” – once you save, this will simply display as snmp-ups$(various)
Port or Hostname: The hostname of the ups, hopefully something simple like ups.mydomain.local

Monitor section:

Leave the Monitor User, Monitor Password, Extra users, and Remote Monitor all blank/default for now - you can change them once the UPS is talking to you. Note I don’t believe it’s important to change the Monitor Password under this configuration, as upsd only listens on the loopback address.

Shutdown and Email sections

I shall leave in your capable hands.

Other Options:

Auxiliary Parameters (ups.conf): This is where you override the SNMP defaults, if desired. By default, NUT will use SNMPv1 and the ‘public’ community. See here for all config options availble for the SNMP driver. You could do something like this, if you want:

community=yaysecurity
snmp_version=3
privProtocol=AES

Auxiliary Parameters (upsd.conf): LISTEN 127.0.0.1 3493

That last line was the magic - this is the bit that makes upsd not only listen on port 161 (whyyyy), but also listen on 3493, which is where the rest of the UPS subsystem expects to find UPS data!

Other interesting tidbits:

Running upsc -l should list all UPS instances configured on your server. Since I’ve got one, called ‘ups’, I see:

# upsc -l
ups

If upsc -l gives you a Connection Refused error, check /usr/local/etc/nut/upsd.conf to make sure it’s listening on 3493. If not, add your auxiliary parameter in TrueNAS and restart the service.

# upsc ups
Error: Connection failure: Connection refused

# cat /usr/local/etc/nut/upsd.conf
LISTEN 127.0.0.1 161
LISTEN ::1 161

# sockstat -4 -l | grep upsd
uucp upsd 99934 6 tcp4 127.0.0.1:161 *:*

Above: upsd is definitely not listening on the expected port

NUT Config files: These live under /usr/local/etc/nut/. The folder is deleted and recreated every time you edit something in the TrueNAS GUI, so if you’re in that folder and change something in the GUI, you’ll need to do the whole cd /usr/local/etc/nut again. No you can’t do cd ../nut, because the whole inode is gone, so the OS doesn’t know where you are anymore.
Testing your UPS: Once upsd is listening on 3493, you should be able to run upsc ups, and it will query your UPS for data. Note that “ups” is your UPS identifier from the “General Options” section, also found as the first line of /usr/local/etc/nut/ups.conf:

# cat /usr/local/etc/nut/ups.conf
[ups]
 driver = snmp-ups
 port = ups.mydomain.local
 desc = 
 pollfreq = 15

# upsc ups
ambient.humidity: 0.00
ambient.temperature: 23.0
battery.charge: 100.00
battery.date: 05/15/2017
battery.packs: 0.00
battery.runtime: 1221.00
battery.runtime.low: 600
battery.voltage: 54.50
device.mfr: APC
device.model: Smart-UPS X 1500
[...]

I think that pretty much covers it. I hope this has been helpful to someone - at very least, I hope it’s helpful to future-me!

It’s been a little while since I’ve worked with UPS management software. I like to think it’s because my day job isn’t in infrastructure anymore, but also to be honest the whole UPS scene is fairly ugly. I recall speaking with an electrical engineer from Schneider a while ago, who jokingly said something like “creating a new UPS or CNC communications protocol is almost a rite of passage for any new engineer”. I can see why… when you’re new, and see something superbly awful, it’s quite compelling to try and fix it! But decades now of dependencies really do add up to be a burden.

Logging DNS queries, for both pfSense and Zentyal server

Sun, 06 Dec 2020 00:00:00 +1300

Logs of your client DNS queries can be a really good tool for incident response. I’ve finally got this implemented but it was much more of a struggle than I expected it to be, so here’s my story!

We’ve got a pfSense firewall running as a DNS fowarder, and a Zentyal server running BIND9 as the authoritative local server. The firewall rules block all UDP/53, so all DNS queries go either directly to the firewall, or (more commonly) to the Zentyal server for resolution. This means we have two places that resolve DNS queries which we want to forward off to our syslog server.

pfSense

Unbound is a super simple DNS forwarder, and the configuration is wonderfully straightforward. It did take me quite a while to figure it out, but in the end it was a simple oversight on my part.

To get pfSense/Unbound to forward DNS queries to your syslog server, simply open the Services -> DNS Resolver page, click ‘Display Custom options’, and add these two lines:

:::ini
server:
log-queries: yes

I spent hours fiddling before I realised I was missing the empty “server:” directive. -__-

This will log all DNS queries that Unbound deals with. These logs go to the standard internal pfSense log; if you want them to be replicated off to an external syslog server, go Status -> System Logs -> Settings. Scroll down to the “Remote Log Servers” section and add your syslog server there. (I’m not covering syslog listener config in this particualar post!)

Zentyal

Yeah this one was harder to figure out. Because Zentyal overwrites the config at every damn opportunity, you can’t just edit /etc/bind/named.conf.options, like what you find in most articles… it may work for a moment, but it’ll be overwritten.

The real answer to editing Zentyal’s config files is outlined at https://doc.zentyal.org/en/appendix-c.html#stubs:

:::bash
cp /usr/share/zentyal/stubs/dns/named.conf.options.mas /etc/zentyal/stubs/dns/named.conf.options.mas

Once you’ve copied the stub into a place where it won’t be clobbered, edit it and zoom right down to the bottom line, which looks like:

:::ini
logging { category lame-servers { null; }; };

This is the line you need to replace. Now, you can go as ham as you want here. I’ve actually used the sample provided by ISC themselves, which you can find at https://kb.isc.org/docs/aa-01526. However, that’s really robust and possibly more than you may need. A bare minimum would probably look like:

:::ini
logging { 
 channel queries_log {
 syslog named;
 print-time yes;
 print-category yes;
 print-severity yes;
 severity info;
 };
};

Note - when I tried to use the ISC sample wholesale, I needed to change a couple small things:

Changed all /var/named/log output destinations to /var/log/named
The lines for category zoneload and category trust-anchor-telemetry need to be removed for Zentyal 6.2’s version of BIND9
I needed to create /var/log/named, and make sure it was writeable by the bind user

I think that covers it, I hope this helps someone! I’ve actually even taken the time today to create a Pull Request in the Zentyal GitHub, we’ll see if it ends up getting merged!

iRedMail, SpamAssassin, and Lynis

Mon, 02 Nov 2020 00:00:00 +1300

I really like iRedMail, and I also really like Lynis.

However, they don’t exactly like each other… or, more accurately, some of Lynis’ recommendations can cause a couple iRedMail components to fail. Today we’re talking about SpamAsassin.

One of the suggestions from Lynis is to turn off the ’execute’ bit on compilers for users who aren’t either the owner or in the owner group (the ‘other’ execute bit). For example:

:::bash
root@server:/# chmod o-x /usr/bin/as
root@server:/# chmod o-x /usr/bin/gcc

Easy peasy! But once you do this, you might start getting the following in your daily iRedMail Cron reports:

:::text
/etc/cron.daily/spamassassin:
/bin/sh: 1: x86_64-linux-gnu-gcc: Permission denied
make: *** [Makefile:346: body_0.o] Error 126
command 'make PREFIX=/tmp/.spamassassin23046Zmmrr9tmp/ignored INSTALLSITEARCH=/var/lib/spamassassin/compiled/5.028/3.004002 >>/tmp/.spamassassin23046Zmmrr9tmp/log' failed: exit 2

There are probably a bunch of ways to fix this. My way, I’m sure, is not the best way… however it was quick and easy, and it worked.

:::bash
root@server:/# chgrp debian-spamd /usr/bin/as
root@server:/# chgrp debian-spamd /usr/bin/gcc
root@server:/# runuser -l debian-spamd -c sa-compile

Look ma, no more errors!

This solution really only works for me because debian-spamd is the only non-root user that calls these compilers. If I had another user which needed to call them, I’d have to come up with a better fix. But for a standalone iRedMail server, this does the trick!

iRedMail: Daily user unknown entries from backup_sogo.sh

Sun, 01 Nov 2020 00:00:00 +1300

If you’ve been running iRedMail for a while, eventually you’ll probably start seeing ‘user unknown’ events in your daily logs:

:::text
* Backup all users' data under /var/vmail/backup/sogo/2020/11/01
<0x0x5574a64c36b0[SOGoCache]> Cache cleanup interval set every 300.000000 seconds
<0x0x5574a64c36b0[SOGoCache]> Using host(s) '127.0.0.1' as server(s)
2020-11-01 09:29:12.784 sogo-tool[29749:29749] user 'abdulm' unknown
2020-11-01 09:29:12.786 sogo-tool[29749:29749] user 'bent' unknown
2020-11-01 09:29:12.786 sogo-tool[29749:29749] user 'brettr' unknown
2020-11-01 09:29:12.786 sogo-tool[29749:29749] user 'catalinar' unknown
2020-11-01 09:29:12.787 sogo-tool[29749:29749] user 'clinth' unknown
2020-11-01 09:29:12.787 sogo-tool[29749:29749] user 'danield' unknown
2020-11-01 09:29:12.787 sogo-tool[29749:29749] user 'dannyn' unknown
2020-11-01 09:29:12.788 sogo-tool[29749:29749] user 'darcyk' unknown
2020-11-01 09:29:12.788 sogo-tool[29749:29749] user 'davidl' unknown
* Compress backup files.

This is because removal of an iRedMail user doesn’t remove the corresponding SOGo user data. You can take care of this with sogo-tool:

:::bash
root@server:/# sogo-tool remove abdulm@<domain.xyz> bent@<domain.xyz> brettr@<domain.xyz>

… etc. Once you’re done, run the backup again to make sure you’ve got them all:

:::bash
root@server:/# /bin/bash /var/vmail/backup/backup_sogo.sh
* Backup all users data under /var/vmail/backup/sogo/2020/11/01
<0x0x563fd0e2b6b0[SOGoCache]> Cache cleanup interval set every 300.000000 seconds
<0x0x563fd0e2b6b0[SOGoCache]> Using host(s) '127.0.0.1' as server(s)
* Compress backup files.

Make the Flex theme center images

Wed, 07 Oct 2020 00:00:00 +1300

Previously I’ve been frustrated that the CSS in the Pelican Flex theme doesn’t centre images. However, I’m only one user with one use-case, so I haven’t raised a pull request.

Instead, I’ve finally gotten around to creating a quick-and-dirty sed string to fix the minified CSS file:

sed -i 's/}img{max-width:100%}/}img{max-width:100%;margin-left:auto;margin-right:auto;display:block}/' style.min.css

I also like my caption text to be italicised and centered, so I extended this a little:

sed -i 's/}img{max-width:100%}/}img{max-width:100%;margin-left:auto;margin-right:auto;display:block}figcaption{font-style:italic;text-align:center}' style.min.css

Run either of those in ./Flex/static/stylesheet. It’ll modify just the img class in the CSS and suddenly all your images will center unless you tell them not to!

ZFS zpool vanishing after upgrading ProxmoxVE 5.4 to 6.2

Sat, 23 May 2020 00:00:00 +1200

Performing a major version upgrade is never pleasant. I’ve been using ProxmoxVE for about ten years now though, and it’s consistently done a fantastic job. Since it’s based on Debian, all the upgrades are done with a simple apt update && apt upgrade, with a variety of steps in the middle to point to new repositories, etc. Nothing out of the ordinary, nothing scary.

So I’ve got two servers - one is an old one, with only a couple testing VM’s residing on it. I go ahead and work through the 5.x to 6.x upgrade guide, and everything Just Works. No problem with that one.

When I moved onto the other server I had a couple different things I noticed. For example, when trying to upgrade Corosync 2.x to 3.x (a prerequisite for the OS upgrade), I noticed it was trying to uninstall things like corosync-pve, which really didn’t seem right.

After comparing dpkg -l contents between the two servers, I learned that corosync-pve is actually just a transitional package - completely replaced by the new corosync package in version 3. And although the server was trying to uninstall corosync-pve, it was then immediately installing corosync… so no problem.

Things continued to be mildly alarming during the upgrade itself, but not actually toooo bad, as all the removed packages were being replaced. This one, for example:

dpkg: pve-libspice-server1: dependency problems, but removing anyway as you requested:
 pve-qemu-kvm depends on pve-libspice-server1 (>= 0.12.5-1); however:
 Package pve-libspice-server1 is to be removed.
 spiceterm depends on libspice-server1 (>= 0.12.2); however:
 Package libspice-server1 is not installed.
 Package pve-libspice-server1 which provides libspice-server1 is to be removed.

(Reading database ... 65255 files and directories currently installed.)
Removing pve-libspice-server1 (0.14.1-2) ...

… was replaced with spiceterm. No worries.

However things went sharply downhill following reboot. All of my VM images were missing! I had a ZFS zpool that should have been at /VM_Local_zpool - that folder existed, and contained an ./images folder, but that was completely empty.

I quickly checked the storage information in Proxmox, and the result was… not good:

… Really, not good. :-|

So I SSH’ed into the server and had a quick look at my list of ZFS stores, and discovered that my dataset was entirely absent:

# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 83.8G 23.8G 96K /rpool
rpool/ROOT 3.65G 23.8G 96K /rpool/ROOT
rpool/ROOT/pve-1 3.65G 23.8G 3.65G /
rpool/data 71.6G 23.8G 71.6G /rpool/data
rpool/swap 8.50G 25.4G 6.85G -

# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 111G 82.1G 28.9G - - 66% 73% 1.00x ONLINE -

A bit of Googling led me to the zfs import command, which resulted in the first piece of good news of the night:

# zpool import

 pool: VM_Local_zpool
 id: 6614373363984244305
 state: ONLINE
 status: Some supported features are not enabled on the pool.
 action: The pool can be imported using its name or numeric identifier, though
	some features will not be available without an explicit 'zpool upgrade'.
 config:

	VM_Local_zpool ONLINE
	 mirror-0 ONLINE
	 wwn-0x50014ee0aeee96ef ONLINE
	 wwn-0x50014ee05998ee25 ONLINE

It exists! And it’s … online? But where? I don’t get it. I went ahead and tried importing it, but got a “directory not empty” error:

# zpool import VM_Local_zpool 
cannot mount '/VM_Local_zpool': directory is not empty

Now, if I’d been paying attention there, I would have realised that zpool list now contained my VM_Local_zpool:

# zfs list
NAME USED AVAIL REFER MOUNTPOINT
VM_Local_zpool 1.16T 611G 1.16T /VM_Local_zpool
rpool 83.8G 23.8G 96K /rpool
rpool/ROOT 3.65G 23.8G 96K /rpool/ROOT
rpool/ROOT/pve-1 3.65G 23.8G 3.65G /
rpool/data 71.6G 23.8G 71.6G /rpool/data
rpool/swap 8.50G 25.4G 6.85G -

From this point, all I had to do was empty /VM_Local_zpool, and try to remount the zpool. But there’s a second trick here.

ProxmoxVE has a pretty clear idea of How Storage Should Look, so every time it scans a storage device, is makes sure there’s an ./images folder in any storage enabled for VM Images. It also makes sure there’s a ./dump folder on any storage enabled as a backup target, and a ./template folder on any storage enabled for ISO images. (Reference: https://pve.proxmox.com/wiki/Storage:_Directory)

So, when I removed that /VM_Local_zpool/images folder, Proxmox just… went ahead and recreated it almost immediately, so I still couldn’t mount the zpool there.

In the end, I managed to get my zpool remounted by pairing the rmdir and mount commands together on the same line:

# rmdir /VM_Local_zpool/images/
# zfs mount VM_Local_zpool
cannot mount '/VM_Local_zpool': directory is not empty
# rmdir /VM_Local_zpool/images/ && zfs mount VM_Local_zpool
# cd VM_Local_zpool/
/VM_Local_zpool# ls
backup	images

… and there we go! My mount point, and all my VM images, are back unscathed!

And just for future reference - I saw a post on the FreeNAS forums that I should the status of the zfs-import-cache service, and enable it if it wasn’t started by default… however it was fine:

# systemctl status zfs-import-cache.service
● zfs-import-cache.service - Import ZFS pools by cache file
 Loaded: loaded (/lib/systemd/system/zfs-import-cache.service; enabled; vendor preset: enabled)
 Active: active (exited) since Sat 2020-05-23 20:46:54 NZST; 3min 26s ago
 Docs: man:zpool(8)
 Process: 1819 ExecStart=/sbin/zpool import -c /etc/zfs/zpool.cache -aN (code=exited, status=0/SUCCESS)
 Main PID: 1819 (code=exited, status=0/SUCCESS)

May 23 20:46:54 ~ systemd[1]: Starting Import ZFS pools by cache file...
May 23 20:46:54 ~ zpool[1819]: no pools available to import
May 23 20:46:54 ~ systemd[1]: Started Import ZFS pools by cache file.

Why are all Split-Tunnel VPN diagrams so ugly?

Thu, 21 May 2020 00:00:00 +1200

Right.

Today I needed to find a diagram of the traffic flow for a split-tunnel VPN. Nothing fancy, just a real simple user-facing diagram to form part of an article.

And friends, there was nothing. Nothing at all. I found proper technical ones from Cisco Meraki; fancy Office365 ones, depicting an ExpressRoute to O365 and tunnelling everything else through the VPN; and another Office365 tunnel with ExpressRoute and a split-tunnel for the rest of the traffic.

Pretty much the closest I came to my needs was this atrocity, via http://blog.soundtraining.net/2013/03/how-to-configure-split-tunneling-on.html:

So I went over to https://www.draw.io and whipped up something a bit prettier. This isn’t marvelous, but it’s way better than I could find anywhere else:

In retrospect, I could have done better - I really should have had the corporate tunnel going through the cloud of the internet. However this works for ne.

If you’d like to use this, feel free. You can even use this fancy URL to open the drawing as an editable vector graphic at Draw.io directly!

Roundcube not loading email contents

Mon, 11 May 2020 00:00:00 +1200

Here’s another quick one! I had a Roundcube instance, chugging along fine.

Then one day it just stopped loading emails! Everything else was all fine - no problems with login etc, but trying to preview an email just game me a sadfaec. Chrome couldn’t load the contents, and told me that the server had refused to connect:

So of course, I can’t just let this go - being able to see the contents of the emails is relatively important! But what the heck was causing it?

I spent some time in Chrome’s developer tools and console, as well as time grepping and tailing log files… this would have all been fixed much earlier if I were running Firefox I think, as it’s a bit more explicit with errors.

So, what was the fix? It was nothing to do with Roundcube, routing, firewalls, or reverse proxies. (Seriously, I checked all of these…)

The problem is that, last week, I’d modified the HTTP Headers on the reverse proxy to include:

 X-Frame-Options ("DENY")

… no good. It all came right when I changed the X-Frame-Options directive to “SAMEORIGIN”. Suddenly, all fixed.

Sheesh!

Renaming a computer object in Active Directory

Sat, 09 May 2020 00:00:00 +1200

Quick post tonight.

You can’t rename a computer object in Active Directory Users and Computers. Even if you change the hostname, the domain object will still have the original distinguishedName from the old device.

However… you can do it in ADSIEdit! Just open ADSIEdit, navigate to the AD container that holds your misnamed object, right click on it, and Rename.

I’ve just done it with a couple test computers on a domain and both worked completely smoothly. I don’t know if this would be safe if the renamed computer were off-site.

WireGuard on Windows - Part 2

Fri, 08 May 2020 00:00:00 +1200

A few days ago I spun up a Windows Dev VM to have a play with WireGuard for Windows.

I didn’t really have a clear goal in mind when I started playing with this - part of it was in trying to create and launch a tunnel without using the GUI. Mostly I was just trying to learn more about a new implementation of a tool that I really like.

If you read my previous article, you’ll recall that I started off trying to do this with just wireguard.exe. Here are some things I’ve discovered:

wireguard.exe does a whole lot of stuff

Wireguard.exe isn’t just a GUI, which I originally thought it was. It’s also the piece of software that shouts out to WinTun to create the interface, as well as the utility that reads the ’extended’ attributes in your .conf file (e.g. the stuff that wg-quick takes care of), as well as the utility that sets up your routes, DNS, etc, etc, etc.

wireguard.exe doesn’t create private/public keypairs

… To do this, you instead need to use wg.exe, which is installed under your System32 folder (so it’s in your path, so it’s accessible anywhere):

PS C:\> wg genkey | tee $ENV:APPDATA\WireGuard.priv | wg pubkey > $ENV:APPDATA\WireGuard.pub

I did figure this out the other day, but I’m reiterating here.

wg.exe can read .conf files – but you don’t really want it to

Last time, I was having trouble reading a .conf file from wg.exe:

PS C:\Users> wg setconf wg0 .\wg0.conf
Line unrecognized: ` ■['
Configuration parsing error

I’m not entirely sure what this was - I’ve since been able to read in a .conf file perfectly fine (for the record this is UTF-8 with Windows CRLF line-endings).

However … my breakthrough kinda sucks.

Wireguard for Windows stores it’s config files in the Windows DPAPI-encrypted vault. This is vastly better than just bunging a file in C:\Users\Blah and hoping for the best. Maybe it’s not perfect - I don’t know much about DPAPI - but it’s a far cry better than nothing.

When you use wireguard.exe to import a tunnel from a .conf file, it will read it in, sanity-check it (mine failed because I accidentally hit the keyboard during copy/pasta, so it rejected the Base64 encoding), and then safely store it away in the DPAPI storage. You can then delete your original .conf file. Just do this, it’s better.

You need both wireguard.exe and wg.exe

OK so here’s the bit that I only fully realised tonight: wireguard.exe is like wg-quick, but it also provides the interface into the Windows network stack and the Windows DPAPI storage of your sensitive conf files. You can’t even run wg set without it, because wireguard.exe is even responsible for creating the IPC Server that interfaces with WinTun.

Honestly the Windows world is so much more complicated than the Linux world*. 🙄

(* some caveats apply)

Conclusion

What’s my plan now?

I’m going to make a thing that:

Installs wireguard.msi silently
Uses wg.exe to create a private/public keypair
Uses that keypair to create a temporary .conf file
Uses wireguard.exe to import that .conf file from some predetermined location
Deletes the .conf file (probably using cipher.exe to scrub it)
Uses wireguard.exe to install the tunnel so it opens automatically on login

I’m hoping that this will create us something like an always-on-VPN connection that can be deployed easily by an IT support person, and require no end-user interaction.

Getting WireGuard on Windows - quietly

Mon, 27 Apr 2020 00:00:00 +1200

So, I mentioned in my post yesterday that I’m trying to get a bit of a quiet installer for WireGuard on Windows. Not that the current one is noisy, but I have a really simple use-case that I want to meet.

I spent some time seeing if I could just extract WireGuard.exe and run that. Nope, of course - it requires WinTun to be installed to facilitate the Layer 3 tunneling.

Next I played with the idea of just deploying WinTun directly, without installing the WireGuard package itself. I can only assume that this would have been super easy if I’d ever built a proper MSI installer before, but in the end it was just way easier to:

WireGuard-amd64-0.1.0.msi /q

… It was still worth the tangential digging, but who would have ever guessed that Jason Donenfeld would have already done things the best way. ¯\_(ツ)_/¯ </s>

Note you’ve got to run the above in an elevated command prompt, since it’s installing drivers. However, the installation is still not quiet - after install, it pops up the WireGuard GUI window!

“That’s annoying,” I think to myself, “can I stop that?” To the installer source code!

OwO … Hwat’s this?

 <!--
 Launch WireGuard.exe after setup complete
 -->
 <CustomAction Id="LaunchApplication" HideTarget="yes" Impersonate="no" Execute="deferred" FileKey="WireGuard.exe" ExeCommand="" Return="asyncNoWait" />
 <InstallExecuteSequence>
 <Custom Action="LaunchApplication" Before="InstallFinalize">(&amp;WireGuardFeature = 3) AND NOT DO_NOT_LAUNCH</Custom>
 </InstallExecuteSequence>

It appears that we can pass through an environment variable to stop the installer from launching the GUI after install!

 .\WireGuard-amd64-0.1.0.msi /q DO_NOT_LAUNCH=True

Aaaand success… now I’ve got WireGuard and WinTun installed, no user interaction, so far completely silent. Next I gotta create a keypair. There aren’t any relevant command-line arguments that I can pass to wireguard.exe, so I went on a hunt for a keypair generator.

I spent an hour or so scratching around various options to find a keypair generator that looked trustworthy. I honestly couldn’t find one… even the one posted on the WireGuard GitHub includes a caveat from Jason.

Until I decided to have a closer look at the installer source code… the very same code I was looking at when I discovered the DO_NOT_LAUNCH property above.

<Component Directory="System64Folder" Win64="yes" Id="Wg64Executable" Guid="d9b494ec-0959-442c-89ad-6aa175acfd03">
 <File Source="..\$(var.WIREGUARD_PLATFORM)\wg.exe" Id="Wg64Executable" />
</Component>

So C:\Program Files\WireGuard\WireGuard.exe appears to be just a shell that calls out to C:\Windows\System32\wg.exe. Oh my god that knowledge would have saved me hours!!

So… here’s how to generate a WireGuard keypair, in Windows, without the GUI. Since system32 is in the path, you can run this from anywhere. The below will create WireGuard.priv and WireGuard.pub in your AppData folder:

PS C:\> wg genkey | tee $ENV:APPDATA\WireGuard.priv | wg pubkey > $ENV:APPDATA\WireGuard.pub
PS C:\> cat $ENV:APPDATA\WireGuard.*
uIGDK+LMDZANniFxrofIpu/hUyezuwCM7qSDVVO+1Gw=
85MxGcWutHJsjFggiZ+J4/vsNYDVEa8zYk53DQZCwyE=

AWESOME. Next step is to zoink that into a handy-dandy config file. (More accurately, next step is to delete that private key that I’ve now posted to the internets)

Right, so it looks like I can’t just create a text-based wg0.conf file like I do in Linux. When I do, I get this error:

PS C:\Users> wg setconf wg0 .\wg0.conf
Line unrecognized: ` ■['
Configuration parsing error

I thought this might be the file encoding, so I tried it in ANSI etc, but then I learned that Wireguard for Windows stores it’s config files in C:\Windows\System32\config\systemprofile\AppData\Local\WireGuard\Configurations\ … and they’re not in plain text. They’re stored with the Windows Data Protection API (DPAPI), so perhaps that’s what wireguard.exe is expecting?

Either way, I’ve been playing with this for quite a long time now, so I’m off to take a break for the night or so. Once I figure out how to create a WireGuard config file for an interface I’ll post a new article. Hope you enjoyed reading so far!

Setting up a Windows Dev VM under ProxmoxVE

Sun, 26 Apr 2020 00:00:00 +1200

Righto, I’ve been using WireGuard for quite a long time now, but I just had my first foray into WireGuard on Windows. I’m trying to roll it into a low-touch VPN deployment sort of thing.

There will be a post about that soon, but this particular post is about setting up the environment for me to play with this stuff… I need a system where I can take snapshots and roll back the drive to a known state. (So I can be sure I’m not missing anything that the installer would do).

I don’t have a Windows machine, so I started off by downloading a Win10 Dev VM from Microsoft. Since I’m running ProxmoxVE, I chose the VMware image, which will come down as a .zip file.

Download the zip file, and extract it to wherever your VM disk images live (mine is /VM_Local_zpool/)… this will save some time when you’re importing the disk.

Next, you simply use the importovf function of qm to import the OVF manifest as a new KVM-based VM in Proxmox!

~ # qm importovf 103 WinDev2004Eval.ovf VM_Images --format qcow2 Formatting '/VM_Local_zpool/images/103/vm-103-disk-0.qcow2', fmt=qcow2 size=136365211648 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16 (100.00/100%)
~ #

That was suspiciously easy…

After importing I couldn’t launch the VM, but I’ve seen this error a dozen times and know who to handle it:

kvm: -drive file=/VM_Local_zpool/images/103/vm-103-disk-0.qcow2,if=none,id=drive-sata0,format=qcow2,cache=none,aio=native,detect-zeroes=on: file system may not support O_DIRECT kvm: -drive file=/VM_Local_zpool/images/103/vm-103-disk-0.qcow2,if=none,id=drive-sata0,format=qcow2,cache=none,aio=native,detect-zeroes=on: Could not open '/VM_Local_zpool/images/103/vm-103-disk-0.qcow2': Invalid argument TASK ERROR: start failed: command '/usr/bin/kvm <-- snip --> -machine 'type=pc'' failed: exit code 1

If you’re here because you googled the above error, your problem is rooted in this particular bit: file system may not support O_DIRECT. Your problem is that Cache setting on your Hard Disk entry (under VM -> Hardware) is set to “Default (No cache)”. Set it to Write Back or Write Through and it’ll launch fine.

Okay, so the VM starts with no network card - I went ahead and just added the VMware vmxnet one that Proxmox offers, since that’ll already have drivers set up on this VMware image. After that it was a few seconds to join the domain, pop the Computer object into the correct OU, run a gpupdate, and fire up my RDP connection using Remmina… and just like that, I’ve got a Windows machine set up to start playing!

Now that I’ve shaved that particular set of yaks, I’m ready to start trying to do what I wanted to do earlier this morning. Let’s see how tomorrow goes…

Using Fail2Ban to insta-block malicious hits

Tue, 07 Apr 2020 00:00:00 +1200

Okay so a few days ago I posted about adopting Caddy.

Of course, anyone who hosts anything knows that accessible services on the internet will start getting hit by scanners within literal moments of being accessible. On the very first night of having my site live on the new VPS, Logwatch informed me of a bunch of attempts to access wp-admin.php. Sigh.

Well, this entire site is built with my favourite static site generator Pelican. That means I don’t have any PHP on my site whatsoever. Why not just… block anyone that tries to open a PHP page at ericlight.com? Fail2Ban to the rescue.

Two things we need: A Fail2Ban filter, and a Fail2Ban jail.

The Filter

nano /etc/fail2ban/filter.d/caddy-php.conf (note, the name of this file must be the same as the identifier in your jail.local file)

[Definition]
failregex = ^<HOST>.*\.php(\s|\?.*)HTTP.*$
ignoreregex =

… that will catch anything in your logs such as:

nn.nn.nn.nn - - [07/Apr/2020:23:20:39 +1200] "GET /wp-admin.php HTTP/2.0" 404 0
nn.nn.nn.nn - - [06/Apr/2020:10:12:24 +1200] "GET /index.php?s=index/%5Cthink%5Capp/invokefunction&function=call_user_func_array&vars%5B0%5D=phpinfo&vars%5B1%5D%5B%5D=1 HTTP/1.1" 404 0
nn.nn.nn.nn - - [06/Apr/2020:12:51:31 +1200] "GET /wp-login.php HTTP/1.1" 404 0

The Jail

nano /etc/fail2ban/jail.local

[caddy-php]
port = http,https
logpath = /var/log/caddy/access.log
enabled = true
maxretry = 1

Note I used maxretry = 1 there. This means if an IP triggers that filter a single time, they’ll be blocked immediately… So you probably don’t want to do this just willy-nilly.

The Result

With those two parts done (and the obligatory service fail2ban reload, of course), you should find that your web server immediately bans any host that tries to load a .php file from your website!

# tail -f /var/log/caddy/access.log /var/log/fail2ban.log
==> /var/log/caddy/access.log <==
1xx.1xx.1xx.11 - - [07/Apr/2020:23:25:55 +1200] "GET /arse.php HTTP/1.1" 404 0
1xx.1xx.1xx.6 - - [07/Apr/2020:23:25:57 +1200] "GET /arse.php?1234 HTTP/1.1" 404 0
1xx.2xx.6x.1xx - - [07/Apr/2020:23:26:03 +1200] "GET /blargh.php HTTP/2.0" 404 0

==> /var/log/fail2ban.log <==
2020-04-07 23:25:55,482 fail2ban.filter [452]: INFO [caddy-php] Found 1xx.1xx.1xx.11 - 2020-04-07 23:25:55
2020-04-07 23:25:55,551 fail2ban.actions [452]: NOTICE [caddy-php] Ban 1xx.1xx.1xx.11
2020-04-07 23:25:58,309 fail2ban.filter [452]: INFO [caddy-php] Found 1xx.1xx.1xx.6 - 2020-04-07 23:25:57
2020-04-07 23:25:58,782 fail2ban.actions [452]: NOTICE [caddy-php] Ban 1xx.1xx.1xx.6
2020-04-07 23:26:03,627 fail2ban.filter [452]: INFO [caddy-php] Found 1xx.2xx.6x.1xx - 2020-04-07 23:26:03
2020-04-07 23:26:04,010 fail2ban.actions [452]: NOTICE [caddy-php] Ban 1xx.2xx.6x.1xx

Great success! You can use the banTime directive to adjust how long these blocks should last for - I’ve got mine set up to block for an hour.

Edit 2020-04-10:

Eh I realised that my RegEx missed these lovely things:

[ip redacted] - - [08/Apr/2020:16:14:10 +1200] "GET /wp-config.php.new HTTP/1.1" 404 0
[ip redacted] - - [08/Apr/2020:16:14:14 +1200] "GET /wp-config.php.old HTTP/1.1" 404 0
[ip redacted] - - [08/Apr/2020:16:14:17 +1200] "GET /wp-config.php.bak HTTP/1.1" 404 0
[ip redacted] - - [08/Apr/2020:16:14:19 +1200] "GET /wp-config.php.backup HTTP/1.1" 404 0
[ip redacted] - - [08/Apr/2020:16:14:22 +1200] "GET /wp-config.php.save HTTP/1.1" 404 0

… so yeah I just changed it to drop anything with .php in the URL. I’ll try to remember not to post any articles with .php in the URL lol.

Credits

Thanks to Phage and Xyphoid for the help in fine-tuning my rusty RegEx!

Moving to the Caddy web server

Sun, 05 Apr 2020 00:00:00 +1300

For the last couple of years I’ve been running this site, as well as my friend’s site (Under The Umbrella) on Nginx. Recently my VPS host decided to do away with their cheapest tier, so instead of doubling my annual cost, I hopped onto https://www.lowendbox.com and found myself a replacement Cheaps McGee VPS to host this.

Well, a major change like that is a great time to learn about something new, so I took the opportunity to get started with Caddy. If you don’t already know about Caddy, it’s a fast, simple, clean web server. It’s written in Go, so it’s both fast and memory safe. And hey, it’s super simple.

I’m not going to go into a whole lot of detail about setting up Caddy - there are enough tutorials out there already, and really I got all the info I needed from the website. But here are some particularly notable bits:

The Caddyfile

This lives in /etc/caddy/Caddyfile:

ericlight.com, www.ericlight.com {

 file_server
 root * /var/www/ericlight.com

 import /etc/caddy/caddy_security.conf

 log {
 output file /var/log/caddy/access.log
 format single_field common_log
 }
}

undertheumbrella.co.nz, www.undertheumbrella.co.nz {

 file_server
 root * /var/www/undertheumbrella.co.nz

 import /etc/caddy/caddy_security.conf

 log {
 output file /var/log/caddy/utu_access.log
 format single_field common_log
 }
}

And, /etc/caddy/caddy_security.conf contains:

header {
 Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
 X-Xss-Protection "1; mode=block"
 X-Content-Type-Options "nosniff"
 X-Frame-Options "DENY"
 Content-Security-Policy "upgrade-insecure-requests"
 Referrer-Policy "strict-origin-when-cross-origin"
 Cache-Control "public, max-age=15, must-revalidate"
 Feature-Policy "accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'self'; camera 'none'; encrypted-media 'none'; fullscreen 'self'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture *; speaker 'none'; sync-xhr 'none'; usb 'none'; vr 'none'"
}

** Update 2022-01-05:** Previously I’d used header / { above; that should have been simply header {. Thanks @JoHoffmann8 for pointing this out! It’s also worth mentioning that Caddy are deprecating the common_log log format, which makes me sad in one way, but I do get it - the Caddy json log file format is far richer, but I liked the easily-ingested syslog format. ¯\_(ツ)_/¯

Ok so here’s the thing. Caddy really seems to implement Python’s ethos of “Batteries Included”. The above contents are enough on their own to:

Host two separate static websites
Offer two subdomains for each of these websites
Manage the entire certificate creation and renewal process from Let’s Encrypt for two sites, plus subdomains.
Get an A+ rating on both SSL Labs and SecurityHeaders.io(!!)
And, of course, zoink all the logs into separate files under /var/log/caddy

Creating a Caddy Service file

UPDATE 2020-05-07: With the release of Caddy 2.0, it appears a regular dpkg -i caddy.deb will take care of creating the caddy.service file

If you’re running Debian, you’ll need to create yourself a service file for systemd, so you can get your server to launch Caddy on boot. I got mine from https://github.com/caddyserver/dist/tree/master/init:

/etc/systemd/system/caddy.service:

# This service file requires the following:
#
# 1) Group named caddy:
# $ groupadd --system caddy
#
# 2) User named caddy, with a writeable home folder:
# $ useradd --system \
# --gid caddy \
# --create-home \
# --home-dir /var/lib/caddy \
# --shell /usr/sbin/nologin \
# --comment "Caddy web server" \
# caddy
#
# 3) Caddyfile at /etc/caddy/Caddyfile that is
# readable by the caddy user
#

[Unit]
Description=Caddy Web Server
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --config /etc/caddy/Caddyfile --environ
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

Preventing scans

Everything above is already pretty secure - Caddy is really good at making security super easy. On top of that, Caddy is the only service hosted on this box, there’s no dynamic code (all raw HTML and CSS, thanks to Pelican), and the only things listening to the internet are SSH and Caddy itself. But even then, I get tired of seeing hundreds of scan reports every day. Fail2Ban to the rescue.

/etc/fail2ban/filter.d/caddy-4xx.conf:

[Definition]
failregex = ^<HOST>.*"(GET|POST).*" (404|444|403|400) .*$
ignoreregex =

/etc/fail2ban/jail.local:

[caddy-4xx]
port = http,https
logpath = /var/log/caddy/access.log
 /var/log/caddy/utu_access.log
enabled = true
banTime = 3600
findTime = 600
maxretry = 5

Fin!

And that’s all! I had another tweak or two to my Pelican Makefile, to point rsync to the right server, but overall that was an incredibly simple process. The Caddy team have done a spectacular job.

Fail2Ban+Nginx (blocking repeated 404's, etc)

Mon, 16 Mar 2020 00:00:00 +1300

After hosting a mail server for a few years, I’ve gotten tired of seeing alllll the 404 attempts in my daily logwatch. Fail2Ban can help here really well, and it turns out to be really easy.

Start with nano /etc/fail2ban/filter.d/nginx-4xx.conf (I’m counting on your running Debian and having things in default locations here), and enter the following:

[Definition]
failregex = ^<HOST>.*"(GET|POST).*" (404|444|403|400) .*$
ignoreregex =

Note: Make sure you use a capital ‘D’ in Definition there.

Now, edit your /etc/fail2ban/jail.conf, and add these lines:

[nginx-4xx]
enabled = true
port = http,https
logpath = /var/log/nginx/access.log
maxretry = 3

Of course, modify your maxretry as desired. Refresh your Fail2Ban rules with service fail2ban restart, and it should now be on the lookout for repeated 4xx errors.

You can confirm it’s working with: fail2ban-client status nginx-4xx:

Status for the jail: nginx-4xx
|- Filter
| |- Currently failed:	2
| |- Total failed:	9
| `- File list:	/var/log/nginx/access.log
`- Actions
 |- Currently banned:	2
 |- Total banned:	2
 `- Banned IP list: 123.232.123.101 123.232.123.102

Success!

Worth mentioning… for the first while I thought it wasn’t working, but I realised I was trying to trigger the rule from on the same LAN, and …

[DEFAULT]
# time is in seconds. 3600 = 1 hour, 86400 = 24 hours (1 day)
findtime = 3600
bantime = 3600
maxretry = 5
ignoreip = 127.0.0.1 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16

… I was hitting it from within the ignoreip range. 😬

And of course, I can’t finish off a blog without giving credit where it’s due. I got this all from GitHub user AysadKozanoglu, here: https://gist.github.com/AysadKozanoglu/1335735272fb3b00a03bd3eea22af818

sa-compile failing during upgrade

Sun, 15 Mar 2020 00:00:00 +1300

I just spent an hour troubleshooting the most ridiculous thing. I guess to help people search, I’d describe this as “dpkg failing at sa-compile in Debian 10” but that’s really not a good picture of what’s happening here…

Setting up sa-compile (3.4.2-1~deb9u3) ...
Running sa-compile (may take a long time)
/bin/sh: 1: x86_64-linux-gnu-gcc: Permission denied
make: *** [body_0.o] Error 126
command 'make PREFIX=/tmp/.spamassassin22062Ifq5yDtmp/ignored INSTALLSITEARCH=/var/lib/spamassassin/compiled/5.024/3.004002 >>/tmp/.spamassassin22062Ifq5yDtmp/log' failed: exit 2
dpkg: error processing package sa-compile (--configure):
subprocess installed post-installation script returned error exit status 25

<-- snip -->

Errors were encountered while processing:
 sa-compile
E: Sub-process /usr/bin/dpkg returned an error code (1)

In my defence, it seemed similar to a recurring issue I’ve had on this server in the past, (which I now promise to address in a separate post). That’s why I ended up on a red herring hunt.

Anyway, I spent a bunch of time trying to troubleshoot dpkg and Python, before realising that a few months ago I’d installed Lynis and worked through a bunch of the hardening recommendations.

… One of which is restricting the execute permissions on /usr/bin/gcc. Which is a symlink to /usr/bin/gcc-6. Which is a symlink to x86_64-linux-gnu-gcc-6.

root@x:/usr/bin# ls -l x86_64-linux-gnu-gcc-6
-rw-r--r-- 1 root root 949016 Feb 15 2018 x86_64-linux-gnu-gcc-6

Yes. I was experiencing problems compiling things because I’d removed execute permissions for all users on the compiler I needed, and I’d spent an hour ignoring the “Permission denied” error that told me where to look.

Fixed with a simple chmod 744 /usr/bin/x86_64-linux-gnu-gcc-6.

FreeNAS finally supports ECC crypto!

Sun, 02 Feb 2020 00:00:00 +1300

Woo! FreeNAS finally supports ECC crypto for it’s SSL certificates in 11.3!

I reported this as a bug almost exactly two years ago, and kept having to do nasty workarounds to make it work every time I installed an update. Very happy that it’s finally sorted and I can use my proper shiny elliptic curve SSL setup! :)

Fortinet SSL VPN Certificate extensions

Sun, 19 Jan 2020 00:00:00 +1300

If you’re setting up a new SSL VPN with certificate authentication, and if you already have an internal PKI, you’re probably considering using your internal certification authority to create the SSL keys and certificates for your VPN clients. Awesome! This is a great and easy way to do this job.

But there’s a dearth of information on the Interwebs regarding what Extended Key Usages you need to have enabled in your Certificate Template for these. There are a heap available, including Email Signing, Timestamping, Code Signing, IPSec SSL things (about 8 of these), and way more options.

Well, your search can end now. There’s only one Extended Key Usage option required for SSL certificate authentication with the Fortinet FortiClient VPN:

Client Authentication

That’s it. That’s the post.

Zentyal 6.0 to 6.1 upgrade getting stuck

Tue, 17 Dec 2019 00:00:00 +1300

So I faced a little challenge with a Zentyal server the other day. I was upgrading from ye olde 6.0 to 6.1, when everything just stopped. I let it sit in the corner for about an hour or so, but it never picked up the thread. All the services were still live, so I logged in to have a look.

(Note, my DNS server is named RIMU; yours may be something else!)

Running ps aux, I discovered this line:

:::bash
sh -c /usr/bin/sudo -p sudo: /var/lib/zentyal/tmp/x2M7gkZVvm.cmd 2> /var/lib/zentyal/tmp/stderr

So, I had a quick look at the contents of that stderr file:

:::bash
# cat /var/lib/zentyal/tmp/stderr 
Password has expired
dns-RIMU@ad.ericlight.com's Password:

And what does that temp .cmd file contain?

:::bash
# cat /var/lib/zentyal/tmp/x2M7gkZVvm.cmd 
kinit -k -t /var/lib/samba/private/dns.keytab dns-RIMU

Running that kinit command indeed prompts for a password reset, but the interesting thing is that samba-tool shows me this password shouldn’t expire:

:::bash
# pdbedit -u dns-RIMU -v | grep change
Password can change: Wed, 31 Oct 2018 21:47:30 NZDT
Password must change: never

… riiiiiiight, that’s a bit interesting. And yet I’m still being prompted to set a new password. I used samba-tool to remind samba that this password shouldn’t expire:

:::bash
# samba-tool user setexpiry dns-RIMU --noexpiry
Expiry for user 'dns-RIMU' disabled.

And now…

:::bash
# pdbedit -u dns-RIMU -v | grep -i change
Password can change: Wed, 31 Oct 2018 21:47:30 NZDT
Password must change: Tue, 19 Jan 2038 16:14:07 NZDT

Now I can run kinit against dns-RIMU perfectly fine, and indeed the Zentyal upgrade succeeded!

StarCraft Remastered broken under Wine

Mon, 09 Dec 2019 00:00:00 +1300

I know this post is only going to be relevant for a couple of weeks, but today I tried to launch StarCraft Remastered under WINE and it crashed.

Pertinent info: StarCraft Version 1.23.2.6926 WINE version 4.20-staging Battle.Net version 1.18.0.11698

Fix was to open the Battle.Net settings -> Game Settings -> “Launch 32-bit client (instead of 64-bit)”

Apparently the 64-bit version has become the default sometime since I last played it (October?!): https://us.forums.blizzard.com/en/starcraft/t/enabling-64-bit/458/7

Inside the Check Point 1180

Sun, 28 Apr 2019 00:00:00 +1200

I’ve got a spare Check Point 1180 here, and I’m fiddling around with it a bit. Learnings so far:

Platform: ARM
CPU / RAM: ARM926EJ-S, 512MB RAM
System board: Marvell RD-88f6281A LE

When a docking station wrecks your breakfast

Sat, 23 Feb 2019 00:00:00 +1300

It’s been a while since I posted, but we bumped into something completely batshit crazy at work that I just had to share.

One of my colleagues was setting up a new computer, and was unable to RDP into the new build. He kept getting the error message “Your session ended because there was a data encryption error. If this keeps happening, ask your admin or tech support for help.”

We’d never experienced this error before, and - although the internet knew about it - there were not very many helpful hits on Google. In fact, all of the reports we found were the result of a bug in Windows 7, and we were only using Windows 10.

In the end, completely weirdly, my colleague tried a different docking station, and it worked! This is absolutely the weirdest thing I’ve ever seen. Both the working and failing docking station had the most recent firmware applied, and they were both flashed from the same image, so it’s not a corrupted download or anything.

Metadata about this issue: Both machines were Windows 10, 1709 build, with all updates installed. The originator machine was an HP Z240 workstation, and the target machine was a Dell 7390, connected to a Dell WD15 docking station. All three items had received fresh firmware updates and all Windows Updates.

New things I didn't know about WireGuard

Thu, 27 Dec 2018 00:00:00 +1300

This is part of my brief series on WireGuard. I’m pretty enamoured with WireGuard and the way it works, and I’ve been using it pretty seamlessly for over a year now. I’ve learned a couple things that weren’t immediately obvious though, so I’m documenting them here.

Easy Provisioning

Samuel Holland mentioned an interesting trinket, in his post at https://lists.zx2c4.com/pipermail/WireGuard/2018-December/003703.html:

"[…] WireGuard will ignore a peer whose public key matches the interface’s private key. So you can distribute a single list of peers everywhere."

You can combine this with wg addconf like this:

Each peer has its own /etc/WireGuard/wg0.conf file, which only contains it’s [Interface] section
Each peer also has a shared /etc/WireGuard/peers.conf file, which contains all the peers
The wg0.conf file also has a PostUp hook, calling wg addconf /etc/WireGuard/peers.conf

It’s up to you to decide how you want to share the peers.conf, be it via a proper orchestration platform, something much more pedestrian like Dropbox, or something kinda wild like Ceph. I dunno, but it’s pretty great that you can just wildly fling a peer section around, without worrying whether it’s the same as the interface.

Setting Private Key from a file

Another piece of learning, courtesy of Samuel Holland, at https://lists.zx2c4.com/pipermail/WireGuard/2018-December/003702.html.

You can read in a file as the Private Key by doing something like:

PostUp = wg set %i private-key /etc/WireGuard/wg0.key

When AppArmor eats your breakfast

Mon, 19 Mar 2018 00:00:00 +1300

I’ve knocked heads with AppArmor a few times now. To be quite honest with myself, I think it’s primarily because I install it, enable all the plugins, and then forget it’s there until there’s a problem.

TL;DR:

On a fully-updated Zentyal 5.0 system running DHCP, the AppArmor profile in /etc/apparmor.d/usr.sbin.dhcpd will prevent isc-dhcp-server from restarting itself after an upgrade.

More search-type words: Zentyal dhcp server doesn’t start again after upgrade. isc-dhcp-server graceful shutdown, but no restart.

This frustration-laden, Google-friendly equivalent of speaking slowly and clearly should illustrate how impossible it was for me to find guidance on this.

To fix:

# aa-complain /etc/apparmor.d/usr.sbin.dhcpd

Okay, now that the TL;DR is finished, here’s the rest of the story:

Unfortunately there’s always a problem with AppArmor eventually.

Even worse, the problem is often silent. I’ll just find that, for example, NONE of my man pages work. Or logrotate will fail. For example, here’s what happens when you enable aa-enforce mode on your friendly resident user manual:

 $ man
 What manual page do you want?
 $ sudo aa-enforce /etc/apparmor.d/usr.bin.man
 Setting /etc/apparmor.d/usr.bin.man to enforce mode.
 $ man
 $man: can't open the manpath configuration file /etc/manpath.config

If I hadn’t enabled aa-enforce immediately before that, what would you have done? You would have gone ahead, made sure that manpath.config existed, made sure it wasn’t corrupted, maybe tried running man as root… maybe even copied a manpath.config file from a known-working computer. But, hidden within dmesg is our old friend:

[149929.763064] type=1400 audit(1521449052.257:111): apparmor="DENIED" operation="open" profile="/usr/bin/man" name="/etc/manpath.config" pid=22633 comm="man" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

And that example is actually better than it used to be - quite a few times in the past four years, I’ve received this:

$ man
$

… from a fully-updated Debian system.

This makes me sad.

Or, in one particular case, ISC-DHCP-SERVER of all things will launch perfectly well, but will fail to restart after a package update, and the first thing I’ll notice is “o hai all the things is broken halp plz”. Something like that.

When you’re troubleshooting DHCP, you’ll probably check systemctl, leases table, maybe /var/log … But when you get around to checking dmesg, you may notice the root of the problem:

From dmesg -T:

[Sat Mar 17 03:44:28 2016] audit_printk_skb: 18 callbacks suppressed

Considering the update occurred at 03:44am, this is probably our problem. I can’t see for sure, but I’m pretty certain those 18 suppressed callbacks are filled with “DENIED” log lines.

The fix is simple:

# aa-complain /etc/apparmor.d/usr.sbin.dhcpd 
Setting /etc/apparmor.d/usr.sbin.dhcpd to complain mode.

And your isc-dhcp-server should now restart gracefully after it’s scheduled upgrades!

Installing PowerChute Network Shutdown on ProxmoxVE

Tue, 27 Feb 2018 00:00:00 +1300

This is going to be one of those posts where I just dump stuff. Power failures are shite, and if your server isn’t talking with your UPS, it won’t shut down gracefully. This is the story of my trying to get APC’s PowerChute Network Shutdown working on a ProxmoxVE 5.1 environment.

Note I opted out of using either nut or apcupsd, because I had a nice fancy Network Management Card (NMC2 / Schneider AP9631) available. Also, I wanted to shut down more than a single server, so a USB or Serial cable wouldn’t do the trick.

Install Java Runtime Environment: apt install openjdk-8-jre
Download PCNS onto your Proxmox server - http://www.apc.com/shop/us/en/categories/power/uninterruptible-power-supply-ups-/ups-management/powerchute-network-shutdown/N-auzzn7
Extract the download: tar -xvf pcns420Linux-x86-64.tar.gz
cd ./Linux_x64
sudo su
./install.sh
When prompted for your Java location, use /usr/lib/jvm/java-8-openjdk-amd64/jre/bin
I didn’t enable SMTP

Results in:

 root@ ~/Linux_x64# ./install.sh 
 ------------------------------------------------------------------
 PowerChute Network Shutdown 4.2.0 for Linux
 Copyright (c) 1999-2016 Schneider Electric.
 All Rights Reserved.
 ------------------------------------------------------------------

 OS=Linux

 Initializing ...

 Press any key to display End User License Agreement
 <--snip-->
 Do you agree to the above license terms? [yes or no]
 yes

 Please enter the installation directory or press enter to install to the default directory (/opt/APC/PowerChute):


 Are you sure you want to install PCNS to /opt/APC/PowerChute [Yes|No]? 
 yes
 PCNS will be installed to /opt/APC/PowerChute

 Please enter java directory if you want to use your system java (example:/usr/local/bin/jre/jre1.8.0_91) or press enter to install the bundled Java:
 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin 

 Checking version of Java ...
 Detected Java Version: 1.8.0.151
 Acceptable version

 openjdk version "1.8.0_151"
 OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-1~deb9u1-b12)
 OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

 JAVA_DIR=/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/

 Copying the installation files ...
 Extracting PCNS files ...
 PCNS is extracted to /opt/APC/PowerChute
 Configuring startup files ...
 Startup script=/etc/rc.d/init.d/PowerChute
 Updating Linux symbolic link ...
 Configure Firewall
 Configuring uninstall script ...
 Setup the m11.cfg file

 Enable SNMP Support [Yes|No]? 
 no
 SNMP Not Enabled

 PowerChute Network Shutdown, v4.2.0
 Copyright (c) 1999-2016, Schneider Electric. All Rights Reserved.
 Startup completed.


 Installation has completed.
 PowerChute Network Shutdown can be accessed through your browser at https://<your_server_ip_address>:6547
 Please complete the configuration wizard so that PowerChute Network Shutdown can protect your server.

Once you’re finished with that, navigate to the new web portal on your server (second-to-last line in the output above), and complete the setup of your PCNS instance!

When you’re configuring PCNS itself, you’ll be prompted for the User Name, Password, and Authentication Phrase for the UPS. These are the user details you use to log into your NMC.

By default, the Authentication Phrase is admin user phrase, but you should totally change this. Assuming you’ve updated the firmware on your NMC, you can set the Authentication Phrase at Configuration -> Shutdown.

Setting up your outlet groups and shutdown times are left as an exercise to the reader. :)

Getting rid of the # format in Flex

Fri, 12 Jan 2018 00:00:00 +1300

Back when I was getting this blog set up, I had a short whinge about the default way that the Flex theme created links to pages.

Specficially, creating a link to “Tuna Patties” (for example), Flex would append the link with an identical stub, such as https://www.ericlight.com/tuna-patties.html#tuna-patties

I thought this was silly, so I found a way to fix it, but never bothered submitting a Pull Request to Alexandre because I figured it was intentional.

Well, it turns out I wasn’t the only one. The good Dmytro Litvinov thought the same, and he created a Pull Request to fix it, almost a whole year ago!

Now, you can avoid this behaviour by simply adding DISABLE_URL_HASH = True to your pelicanconf.py!

Fix Dell Keyboard Backlight under Debian

Thu, 11 Jan 2018 00:00:00 +1300

My personal laptop is an old Dell Latitude E6410. One of the things I actually love about it, is the fact that the keyboard has a backlight.

However, because Linux, sometimes that backlight just stops working. When this happens, you’ll see error messages in dmesg, such as:

dell_wmi: Unknown key with type 0x0011 and code 0x01e2 pressed

If this happens to you, just run this command:

$ sudo echo 7 > /sys/devices/platform/dell-laptop/leds/dell\:\:kbd_backlight/brightness

You can echo a higher number if you want your keyboard brighter, but I found 7 to be more than bright enough.

Mikrotik RouterOS - incorrect SNMP status for idle WLAN interfaces

Sun, 07 Jan 2018 00:00:00 +1300

While trying to monitor the status of one of our wireless access points, I discovered that RouterOS returns an incorrect SNMP status code for the wlan interfaces when there are no clients connected to the interface. This is at least present in RouterOS v6.40.3, on a Mikrotik cAPlite (RBcAPL-2nD).

Specifically, when the wifi interface is up, but has zero connected clients, the OID returns an SNMP value of 2 (“down”). This status is incorrect - the interface isn’t down, it’s merely waiting for an external connection.

The specific OID is 1.3.6.1.2.1.2.2.1.8.2 (iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifOperStatus - interface #2 in my case)

The applicable RFC says, “When ifAdminStatus changes to the up state, the related ifOperStatus should do one of the following: […] Change to the dormant state if and only if the interface is found to be operable, but the interface is waiting for other, external, events to occur before it can transmit or receive packets” (https://www.ietf.org/rfc/rfc2863.txt, section 3.1.13)

I notice this issue was logged on the forums about 6 years ago, at https://forum.mikrotik.com/viewtopic.php?t=51332.

I’ve raised this issue with Mikrotik support, and apparently it’s been added to their to-do list, but I’m not sure it’ll ever be done. In the meantime, we’ve just had to disable up/down monitoring of that specific interface, because it’s so misleading.

When the problem is DNS: FreeNAS and NFS

Sun, 07 Jan 2018 00:00:00 +1300

I discovered a while ago that NFS requires DNS to function correctly: https://redmine.ixsystems.com/issues/4983

That’s somewhat annoying, because when your DNS server goes down (and your backups are stored on your FreeNAS server and accessed over NFS), is precisely the time when you really want your backups to be accessible.

However, turns out it doesn’t aaaaactuallyyyyy need DNS… it needs name resolution. Specifically, FreeNAS just needs to be able to resolve it’s own hostname. Cue the handy-dandy hosts file.

Under Network -> Global Configuration -> Host name data base, add your details:

192.168.88.5 backupserver backupserver.local backupserver.ad.mydomain.com

If you have a secondary IP address, duplicate the above line and replace the IP as appropriate.

Obviously, replace the hostname and your internal domain name with whatever you use. However, be aware that the .local FQDN is actually required, even if you use a different domain name.

Flashing a Samsung S5 G900I back to stock

Tue, 19 Dec 2017 00:00:00 +1300

I got a second-hand Samsung Galaxy S5 for my mum yesterday (a G900I model, from Telstra), and I spent some time getting it ready.

Firstly, I downloaded the most-recent Telstra firmware image. At time of writing, that’s G900IDVU1CQJ2. Telstra appears to be the only carrier still releasing stock firmware for this phone, so I downloaded the Telstra version even though we’re connecting to Spark New Zealand. This means the phone will at least be running the most up-to-date baseband and modem firmware.

Note - if you try this, and find that the recent (international) version doesn’t work properly with your local carriers, download the older (local) package, and flash the modem.bin and the NON-HLOS.bin packages from the local one instead.

Flashing the new firmware required a little thinking. I’m 99% sure that I did this the hard way, but basically:

Download the newest version of Heimdall from Github
Back last year I had a post that mentioned editing CMakeLists.txt - ignore that now.

Build the software (assuming you extracted the zip instead of using git clone), with:

 cd Heimdall-master
 mkdir build
 cd build
 cmake -DDISABLE_FRONTEND=ON -DCMAKE_BUILD_TYPE=Release .. 
 make

Once I’d built Heimdall, I unzipped the firmware into a convenient place, and I examined the PIT file from the phone:

./heimdall print-pit

This descripts the partition table on the phone. In particular, it tells you which files in your firmware package should be uploaded to which partitions. I ended up with the following Heimdall flash line:

./heimdall flash --APNHLOS NON-HLOS.bin --MODEM modem.bin --SBL1 sbl1.mbn --DBI sdi.mbn --ABOOT aboot.mbn --RPM rpm.mbn --TZ tz.mbn --BOOT boot.img --RECOVERY recovery.img --SYSTEM system.img.ext4 --CACHE cache.img.ext4 --HIDDEN hidden.img.ext4

I know I could have only flashed the modem.bin and NON-HLOS.bin files, but I’m not sure what I would have missed by skipping all the other partitions.

After the phone was flashed up to the current Stock firmware, I made sure it booted correctly, and then went ahead and converted the phone to my beloved LineageOS, happy in the knowledge that all the “other bits” of the firmware were all updated.

Making MindTouch's Dekiwiki work on Debian Stretch

Sun, 17 Sep 2017 00:00:00 +1200

We had accumulated a certain amount of technical debt, due to a Wiki solution that was selected a few years ago: Dekiwiki, by MindTouch. Unfortunately a few months after implementation, MindTouch Core (which Dekiwiki builds on) was well-and-truly deprecated back in 2013.

It all happened before my time, but it seems as if Dekiwiki came as a pre-built VMware Appliance, based on Debian Etch (Debian 4.0; released in 2007). After giving our Dekiwiki environment some serious side-eye for a while, I finally decided to get my hands dirty and try to upgrade it.

Jedd, over at Jeddi.org, wrote a really helpful post back in 2015 about his experiences updating the ageing version of Debian that the VM is based on. The whole Dekiwiki image is held together by sticky-tape, but that post is really helpful in getting Etch upgraded to Wheezy.

Here are a couple additional things I’ve had to do (please, please look at Jedd’s post above - he covers a whole lot that I’m not covering here):

Upgrading Etch -> Lenny:

Follow the guide at Jeddi.org above to get to Lenny. I can’t remember much failing for that upgrade step, but I remember something happened.

From Lenny -> Squeeze:

DNS Breaks. You’ll notice that SSH takes aaaages to log into, and other things will start being slow and complaining.

Resolve this by editing the ‘hosts’ line in /etc/nsswitch.conf:

hosts: files dns

Originally this reads hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4

Note I’m not actually sure this happened between Lenny and Squeeze; it may have been from Squeeze to Wheezy. *shrug*

From Squeeze -> Wheezy:

MySQL breaks. When visiting the wiki, you’ll be informed that it couldn’t access the backend, and that the API might be warming up. However, you’ll notice that the MySQL service fails to start. When you run ‘mysqld’ from command line, you’ll see it’s missing a folder.

Resolve this by running mkdir /var/lib/mysql-files ; chown mysql:mysql /var/lib/mysql-files

From Wheezy -> Jessie (upgrade-only):

Apache breaks. You’ll notice Apache starts serving just the contents of /var/www. This is because the Apache config file now looks for *.conf in /etc/apache2/sites-enabled; of course, the dekiwiki config file doesn’t have the .conf ending.

Resolve this by running mv /etc/apache2/sites-enabled/dekiwiki /etc/apache2/sites-enabled/dekiwiki.conf

From Wheezy -> Jessie (dist-upgrade):

Mono breaks. If you upgrade mono, Dekiwiki will fall back into unlicensed mode. The licensing engine will no longer be able to interpret license files correctly. You’ll see the error “Server license validation failed. The license signature is not valid.”

Resolve this by holding back the mono packages before you run a dist-upgrade:

apt-mark hold libapache2-mod-mono libmono-2.0-1 libmono-2.0-dev libmono-accessibility2.0-cil libmono-accessibility4.0-cil libmono-bytefx0.7.6.2-cil libmono-c5-1.1-cil libmono-cairo2.0-cil libmono-cairo4.0-cil libmono-cecil-private-cil libmono-codecontracts4.0-cil libmono-compilerservices-symbolwriter4.0-cil libmono-corlib2.0-cil libmono-corlib4.0-cil libmono-cscompmgd8.0-cil libmono-csharp4.0-cil libmono-custommarshalers4.0-cil libmono-data-tds2.0-cil libmono-data-tds4.0-cil libmono-data2.0-cil libmono-db2-1.0-cil libmono-debugger-soft2.0-cil libmono-debugger-soft4.0-cil libmono-dev libmono-firebirdsql1.7-cil libmono-getoptions2.0-cil libmono-http4.0-cil libmono-i18n-cjk4.0-cil libmono-i18n-mideast4.0-cil libmono-i18n-other4.0-cil libmono-i18n-rare4.0-cil libmono-i18n-west2.0-cil libmono-i18n-west4.0-cil libmono-i18n2.0-cil libmono-i18n4.0-all libmono-i18n4.0-cil libmono-ldap2.0-cil libmono-ldap4.0-cil libmono-management2.0-cil libmono-management4.0-cil libmono-messaging-rabbitmq2.0-cil libmono-messaging-rabbitmq4.0-cil libmono-messaging2.0-cil libmono-messaging4.0-cil libmono-microsoft-build-engine4.0-cil libmono-microsoft-build-framework4.0-cil libmono-microsoft-build-tasks-v4.0-4.0-cil libmono-microsoft-build-utilities-v4.0-4.0-cil libmono-microsoft-build2.0-cil libmono-microsoft-csharp4.0-cil libmono-microsoft-visualc10.0-cil libmono-microsoft-web-infrastructure1.0-cil libmono-microsoft8.0-cil libmono-npgsql2.0-cil libmono-npgsql4.0-cil libmono-opensystem-c4.0-cil libmono-oracle2.0-cil libmono-oracle4.0-cil libmono-peapi2.0-cil libmono-peapi4.0-cil libmono-posix2.0-cil libmono-posix4.0-cil libmono-rabbitmq2.0-cil libmono-rabbitmq4.0-cil libmono-relaxng2.0-cil libmono-relaxng4.0-cil libmono-security2.0-cil libmono-security4.0-cil libmono-sharpzip2.6-cil libmono-sharpzip2.84-cil libmono-sharpzip4.84-cil libmono-simd2.0-cil libmono-simd4.0-cil libmono-sqlite2.0-cil libmono-sqlite4.0-cil libmono-system-componentmodel-composition4.0-cil libmono-system-componentmodel-dataannotations4.0-cil libmono-system-configuration-install4.0-cil libmono-system-configuration4.0-cil libmono-system-core4.0-cil libmono-system-data-datasetextensions4.0-cil libmono-system-data-linq2.0-cil libmono-system-data-linq4.0-cil libmono-system-data-services-client4.0-cil libmono-system-data-services4.0-cil libmono-system-data2.0-cil libmono-system-data4.0-cil libmono-system-design4.0-cil libmono-system-drawing-design4.0-cil libmono-system-drawing4.0-cil libmono-system-dynamic4.0-cil libmono-system-enterpriseservices4.0-cil libmono-system-identitymodel-selectors4.0-cil libmono-system-identitymodel4.0-cil libmono-system-ldap2.0-cil libmono-system-ldap4.0-cil libmono-system-management4.0-cil libmono-system-messaging2.0-cil libmono-system-messaging4.0-cil libmono-system-net4.0-cil libmono-system-numerics4.0-cil libmono-system-runtime-caching4.0-cil libmono-system-runtime-durableinstancing4.0-cil libmono-system-runtime-serialization-formatters-soap4.0-cil libmono-system-runtime-serialization4.0-cil libmono-system-runtime2.0-cil libmono-system-runtime4.0-cil libmono-system-security4.0-cil libmono-system-servicemodel-discovery4.0-cil libmono-system-servicemodel-routing4.0-cil libmono-system-servicemodel-web4.0-cil libmono-system-servicemodel4.0-cil libmono-system-serviceprocess4.0-cil libmono-system-transactions4.0-cil libmono-system-web-abstractions4.0-cil libmono-system-web-applicationservices4.0-cil libmono-system-web-dynamicdata4.0-cil libmono-system-web-extensions-design4.0-cil libmono-system-web-extensions4.0-cil libmono-system-web-mvc1.0-cil libmono-system-web-mvc2.0-cil libmono-system-web-routing4.0-cil libmono-system-web-services4.0-cil libmono-system-web2.0-cil libmono-system-web4.0-cil libmono-system-windows-forms-datavisualization4.0-cil libmono-system-windows-forms4.0-cil libmono-system-xaml4.0-cil libmono-system-xml-linq4.0-cil libmono-system-xml4.0-cil libmono-system2.0-cil libmono-system4.0-cil libmono-tasklets2.0-cil libmono-tasklets4.0-cil libmono-wcf3.0-cil libmono-web4.0-cil libmono-webbrowser2.0-cil libmono-webbrowser4.0-cil libmono-webmatrix-data4.0-cil libmono-windowsbase3.0-cil libmono-windowsbase4.0-cil libmono-winforms2.0-cil libmono0 libmono2.0-cil mono-2.0-devel mono-2.0-gac mono-4.0-gac mono-apache-server2 mono-common mono-csharp-shell mono-dmcs mono-gac mono-gmcs mono-mcs mono-runtime mono-utils mono-xbuild

Note: Some of these can certainly be unheld. I haven’t yet gone through and identified exactly which component is breaking on upgrade. If you put in the effort to determine the culprit, please let me know.

That’s all! You should now have Dekiwiki running on Debian Stretch! I’m planning to upgrade it to Debian Buster in the next week or two.

Compiling Heimdall without installing Qt

Wed, 13 Sep 2017 00:00:00 +1200

During my recent Wrecked-phone Saga, I had some trouble flashing my firmware. My phone was broken at the time, so I couldn’t enable ADB Debugging - therefore adb wasn’t an option. OEM Bootlock was on, so fastboot wasn’t an option. My Recovery bootloader was also broken. I thought my phone was bricked.

That’s until I remembered Heimdall.

Heimdall is specifically a tool for working with the partitions on Samsung phones - other visitors need not apply. It works a treat for my Samsung Galaxy S5 (kltedv) though.

Heimdall, however, is old. The version in the Debian Sid repository is 1.4.1, which dates back to 2015. And when we have old versions, we also have incompatibilites! Such as Heimdall 1.4.1’s incompatibility with newer versions of Samsung devices … such as my precious SGS5.

Happily, Heimdall is open source, so I was able to download and compile a new version that overcomes the problem. And that’s how I found myself staring at the screen and looking at this:

That is Yuck.bat

I really didn’t want to install 55 packages just to build this. The majority of missing packages were required to build the GUI section of the tool, which I didn’t want to use anyway. What if I could … build Heimdall without building the GUI??

And of course, that’s the title of this article, so here’s how to do it!

EDIT 2017-12-19: The stuff below is no longer necessary. For better instructions, have a look at my article about flashing a Samsung G900I back to stock.

Clone the git repository, with git clone https://github.com/Benjamin-Dobell/Heimdall.git
Edit the CMakeLists.txt file: nano Heimdall/CMakeLists.txt

Delete the last four lines of CMakeLists.txt:

 cmake_minimum_required(VERSION 2.8.4)

 set(CMAKE_MODULE_PATH
 ${CMAKE_SOURCE_DIR}/cmake
 ${CMAKE_MODULE_PATH})

 project(Heimdall)

 set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin)

 option(DISABLE_FRONTEND "Disable GUI frontend" OFF)

 add_subdirectory(libpit)
 add_subdirectory(heimdall)

You’re done! Go ahead and build according to the instructions.

When Encryption Attacks!

Tue, 12 Sep 2017 00:00:00 +1200

So… I wrecked my phone last night. :-(

Android’s phone encryption feature has been around for aaaages, so I was confident when I started the encryption process last night. Unfortunately, something went wrong, and last night I discovered that everything was toast.

After a bit of digging, I found someone else with exactly the same problem!

“unable to boot into my phone as it sits at the boot screen. When I try to boot into TWRP, it asks for my password. […] it appears to decrypt the partition and mount, but then while loading, TWRP shows a continuous stream of:

E:Error parsing XML file

errors until it just restarts.”

Unfortunately, nobody responded to my own personal denvercoder9.

Long story short, I lost everything. But for a while, I thought my entire phone was bricked. The phone wasn’t accessible in either Recovery mode (due to the XML parsing catastrophe) or Download mode - it wasn’t accessible either via adb or via fastboot.

Eventually my mate @rendition pointed out that I could just boot into Recovery, then cancel the decryption step and wipe my phone from there. That allowed me to start again from a clean slate.

And I encrypted my phone right away, with no trouble at all!

Firmware update on an APC AP9630 NMC2

Sat, 02 Sep 2017 00:00:00 +1200

I’ve spent a little while working with the APC Network Management Cards now, and firmware updates are a total pain.

The biggest issue is that the UPS power outlets need to be powered off in order to flash the firmware, otherwise there’s a terrifyingly-high chance that the NMC (a ~$700 card) will completely shit itself, and die permanently. Aside from that, I’ve never managed to get updates working properly from the web interface.

Fortunately, the card is hot-pluggable, so if you have a spare UPS hanging around, you can unplug the NMC from each UPS in your fleet, throw it into your spare, and flash the firmware from there. Fast, easy, and simple… as long as you have that all-important spare.

Download the Firmware

First, download the appropriate firmware for your device. At time of writing, the link for the SmartUPS 6.5.0 NMC firmware can be found here, but if it’s been a while you should find a new version at http://www.apc.com/us/en/tools/download/.

It comes down as an EXE file, but use 7-zip to extract the archive somewhere convenient. You only need the three .bin files:

apc_hw05_bootmon_108.bin
apc_hw05_aos_650.bin
apc_hw05_sumx_650.bin

Flashing the Firmware

Flashing the actual firmware is easy, but you do need to be careful of the order. The bootmon package goes first, then the AOS package, and then the sumx package.

Be aware: the NMC will reboot immediately after each file is uploaded. As a result, you need to reconnect after each upload, and reissue the BIN command.

Make sure you know the username and password (default credentials are apc/apc), and that the FTP server is enabled. Then, go ahead and ftp to the NMC on it’s IP address.

$ ftp
ftp> open x.x.x.x
Connected to x.x.x.x.
Name (eric): apc
Password: apc
230 Login successful.
Remote system type is UNIX.
Using text mode to transfer files.
ftp> bin
Using binary mode to transfer files.

Make sure you send the ‘bin’ command. I also use the hash command so I can watch the transfer status, but it’s unnecessary.

Once you’ve connected, go ahead and upload the bootmon file:

ftp> put apc_hw05_bootmon_108.bin
ftp> close

It’ll only take a few seconds to upload the file, and you won’t see any notification from the FTP server. But if you’re paying attention to the card itself, you’ll see the NMC network connection go down, and then start flashing. This is the NMC installing the new firmware package.

Once the connection is up again, your FTP session will be broken. Go ahead and reconnect; remember to reissue the BIN command:

ftp> put apc_hw05_bootmon_108.bin
ftp> close
<...snip...>
ftp> open x.x.x.x
<...snip...>
ftp> bin
Using binary mode to transfer files.

Now upload the AOS file. When the NMC reboots again, go ahead and repeat the cycle to upload the sumx file.

Once the NMC reboots for the third time, you should be done!

Thanks

Thanks to Angela from APC - she’s been around for years, and as far as I can see she’s THE leading customer support person for the entire NMC firmware suite. You can find her post on this process here at the APC forums.

Respecting Amavis' "Banned Extensions" setting

Sun, 13 Aug 2017 00:00:00 +1200

I’ve been dabbling a little bit with iRedMail, mostly just to have a play with a mail server, but also to see what’s involved in mail security. iRedMail is a package that pulls together Postfix as an MTA, Dovecot as a POP3 & IMAP server, SOGo for ActiveSync, Roundcube for Webmail, SpamAssassin for spam protection, and ClamAV for virus scanning.

Okay I have no idea why I have to write this, but apparently it’s a thing.

Amavis has a list of banned file extensions. In Debian, they live in /etc/amavis/conf.d/20-debian_defaults, and /etc/amavis/conf.d/50-user, and are set in the $banned_filename_re variable. THIS MAKES PERFECT SENSE.

But of course, there’s always something that doesn’t make sense, and that is the fact that there is a SQL backend (at least in the environment created by iRedMail), and settings in here take precedence over the Amavis config files somehow.

And even more bizarrely, there exists in this SQL environment, a policy setting entitled “banned_files_lover”, which was set to “Y”. I shit you not. My only hope is that this only defaults to “Y” for postmaster.

To fix this, you need to hop into the database, and update the appropriate column in the policy table:

$ mariadb
MariaDB [none]> \u amavis
MariaDB [amavisd]> update policy set banned_files_lover="N";

And now, Amavis will obey your file extension filters!

Thankfully, I found this information at http://www.iredmail.org/forum/topic13147-iredmail-support-amavisd-passed-but-setup-at-ddiscard.html – I never would have found it otherwise!

Making Amavis work with ESET Antivirus

Sat, 12 Aug 2017 00:00:00 +1200

But of course, ClamAV has shown disappointing performance, and it would be really nice to use something more… commercially suitable.

To tie together mail receipt and scanning, iRedMail uses Amavis (strictly speaking, ‘amavisd-new’). Amavis uses ClamAV by default, but it comes with a bunch of configuration blocks to bring together other antivirus applications.

But although amavisd-new is stable and still maintained, some parts of it are really old. In particular, many of these av-scanner config blocks are… uhh… “deprecated”. There’s one particular entry for ESET that is dated 2002 - things have changed a lot in the last fifteen years. *shudder*

So, with the help of some documentation, I managed to piece together a code block that works:

['ESET File Security for Linux',
 ['/opt/eset/esets/sbin/esets_scan','esets_scan'],
 '--subdir --unsafe --unwanted --clean-mode=strict {}',
 [0,10,100],[1,50],
 qr/threat="([^"]+)"/m
],

That would be the end of the story, except I also had to hunt around to find out the right place to put my beautiful codeblock into. It turns out that Debian’s Amavis config structure is quite different to the CentOS config that is most-frequently mentioned in the iRedMail forums. I spent a lot of time playing with /etc/amavis/conf.d/15-av_scanners, and nothing seemed to work. Eventually I found out that Debian features a /etc/amavis/conf.d/50-users file that overwrites the settings from 15-av_scanners. Finally I had progress!

Somewhere around line 154 in /etc/amavis/conf.d/50-users, you’ll find an @av_scanners codeblock. I deleted the ClamAV section in there, and replaced it with the ESET codeblock above. I left the ClamAV settings in the @av_scanners_backup section, because Amavis will fall back to that if ESET fails.

That seems to be all! At least, it works with the EICAR anti-malware test file.

RingZer0team CTF - Challenges 86, 87, and 88

Thu, 13 Jul 2017 00:00:00 +1200

There are a bunch of fantastic Capture The Flag security challenges on RingZer0Team.com. I’ve been working through some of these for a wee while now, and with the New Zealand Cyber Security Challenge coming up again soon, I thought I’d get back into some of them.

Challenge 86 (“1/3 Do not waste the environment”, under the Forsensic Challenges) is one of a series of challenges where you need to dig through some provided data to find the flag.

I started by downloading the ‘forensic bundle’, which was just a large zip file. The first challenge was to figure out the contents of that zip file. The file name was just a jumble of characters, and there was no extension. Running file on it just returned the file type “Data”… not very enlightening, but head uncovered the string “VBOX” in the file… okay, it’s VirtualBox file.

Because I didn’t have Virtualbox installed, I spent a bit of time digging through the actual Vbox file itself, trying to see if there’s a flag already in there. I used strings | egrep -i flag-...., to try to find any flags embedded in the file itself. I didn’t find any flags, but I did discover that the computer name itself is “FLAG-PC”. Very clever. :-|

Update, many hours later:

Okay this totally cooked my bacon. After hours of playing with VirtualBox and playing with files, I finally gave up and Googled the name of the downloaded file.

Turns out I was REALLY close. The flag IS embedded in the actual file itself, but the format is different for the first time. The correct grep string would have been egrep -i '.?f.?l.?a.?g.?-'.

Real frustrated by how close I was, I’ll have to expand my searches in the future. Hat-tip to @professormahi and his GitHub page.

Once I understood that, I also discovered the flag for challenge 87 (“2/3 Did you see my desktop?”), in the same manner.

I haven’t managed to get the flag for Challenge 88 (“3/3 Suspicious Account Password”) yet, but I do know what it is. I can see (by grepping for “Visited”) a bunch of visits to http://www.forensicswiki.org/wiki/Tools:Memory_Imaging… this thing is a memory dump. That explains now why it doesn’t work as a VirtualBox disk.

Next time I sit down here, I’ll find visit Forensics Wiki, and start digging through the memory dump for the passwords.

Update two

I decided not to put this up as a separate post, because once I knew what I was looking at, it was pretty easy. Again though, I relied on @professormahi’s work - but now thanks to him I have a little bit of experience with Volatility!

First, we get Volatility to scan the image and see what it’s dealing with:

$ volatility -f vm.vmdk imageinfo
Volatility Foundation Volatility Framework 2.6
INFO : volatility.debug : Determining profile based on KDBG search...
 Suggested Profile(s) : Win7SP1x86_23418, Win7SP0x86, Win7SP1x86
 AS Layer1 : IA32PagedMemory (Kernel AS)
 AS Layer2 : FileAddressSpace (/home/eric/Downloads/86/vm.vmdk)
 PAE type : No PAE
 DTB : 0x185000L
 KDBG : 0x82920be8L
 Number of Processors : 1
 Image Type (Service Pack) : 0
 KPCR for CPU 0 : 0x82921c00L
 KUSER_SHARED_DATA : 0xffdf0000L
 Image date and time : 2014-03-09 20:57:55 UTC+0000
 Image local date and time : 2014-03-09 13:57:55 -0700

Now, knowing the profile that we need to use, we can use it to just… dump the password table:

$ volatility -f vm.vmdk --profile Win7SP1x86 hashdump
Volatility Foundation Volatility Framework 2.6
Administrator:500:aad3b435b51404eeaad3b435b51404ee:31d6cfe0d16ae931b73c59d7e0c089c0:::
Guest:501:aad3b435b51404eeaad3b435b51404ee:31d6cfe0d16ae931b73c59d7e0c089c0:::
flag:1000:aad3b435b51404eeaad3b435b51404ee:3008c87294511142799dca1191e69a0f:::

And then, we grab that NTLM hash and pop it into an online NTLM decryptor service, and we’re away!

Staying Secure in Business

Wed, 28 Jun 2017 00:00:00 +1200

A friend asked a question today on Facebook. I started writing a reply, and it turned into a 700-word essay. Hate it when that happens…

The question was:

“If large multi-international companies are getting hacked in Europe who have millions if not billions of $$ and capacity to protect their IT systems - how can small/micro businesses protect their IT platforms and systems?”

Firstly - nobody is safe from an APT (“Advanced Persistent Threat”). The recent NotPetya outbreak was an APT - malicious actors hacked a Ukrainian firm that produced accounting software, and used their software’s update facility to literally deliver malicious code specifically to the users of this particular software package (“M.E. Doc”). Nothing’s keeping folks safe from that level of sophistication.

However, the exploit by which NotPetya spreads was actually patched by Microsoft in March. If people had updated their internet-connected devices, this virus would have been limited to only users of M.E. Doc.

Aside from that though, there’s actually a disappointingly simple answer to this. Security is actually relatively within reach, if we all follow these simple steps:

1) Update. Everything. At every opportunity.

That includes Windows Updates, but also Adobe Reader (annoying!), Java (double annoying!), anti-virus engines, etc. This is absolutely the top priority, and the cybersecurity industry has been trotting this line out for years, but people just don’t. Because annoying.

Real life incident: June 22, 2017 - Honda falls prey to the WannaCry worm, 37 days after release (four months after the patch was released), due to poor patching hygiene.

See http://thehackernews.com/2017/06/honda-wannacry-attack.html

2) Backup everything important.

The more important it is, the more places you should keep it. You should have at least three copies of every important file. THIS IS ANNOYING, but if you don’t do it, data loss will hit you. Something like Crashplan (a low-cost service that sends backups to the cloud), plus a regular (weekly?) copy to a read-only media (e.g. a DVD), should be enough for a small business.

Real life incident: Feb 1, 2017 - GitLabs suffers a major outage after data was accidentally deleted, followed by a sequence of discoveries that none of their five layers of backup or replication techniques “are working reliably or set up in the first place”.

See https://techcrunch.com/2017/02/01/gitlab-suffers-major-backup-failure-after-data-deletion-incident/

3) Use next-generation anti-virus software.

Many of the AV products these days include very clever ways of analysing the behaviour of unknown software, and literally just rolling back all the changes if the AV decides it’s not trustworthy. Webroot is a good example of this, at a very affordable price.

Real life incident: April 23, 2017 - the Adylkuzz virus started quietly spreading around the internet, and setting up Monero cryptocurrency mining operations on infected PC’s. Because the payload wasn’t destructive, it flew under the radar and was missed by many antiviruses for weeks.

See https://www.proofpoint.com/us/threat-insight/post/adylkuzz-cryptocurrency-mining-malware-spreading-for-weeks-via-eternalblue-doublepulsar

4) Be alert, cautious, and sufficiently paranoid.

The last - and hardest - step is to maintain a healthy sense of distrust; sometimes referred to as “sufficient paranoia”. When you receive an email, ask yourself: is this email in-character for this person? Is the spelling what I’d expect? Is it appropriate? Is it asking me to take an action (even if that action is just opening a file)? If so, is it expected, or is it out of the blue? If it’s out of the blue, why? What’s the story behind that?

Real life incident: Sept 30, 2011 - I received an email from a lady I’d once interviewed for a job. It was a distraught email saying she was stuck in Wales, had been mugged, lost all her money, and needed a brief loan until she got home. I wrote her an email (not a reply), telling her email had been hacked; lo and behold, I got a reply! “It’s me… this is for real, i have checked with the consulate but there is nothing really working out, most important is i don’t have enough money on please, please i need you to loan me some, i can pay you back once i get home.”

I ended up finding her cellphone number and sending her a message - she was at home, safe and sound.

See https://blog.malwarebytes.com/cybercrime/2014/08/email-hijack-leads-to-i-was-robbed-send-me-money-scam/ – this was exactly the wording I received.

Geez, that was a long Facebook post Eric.

It really was, and a few minutes afterwards, this tweet came to my attention:

https://twitter.com/josephfcox/status/879982828418719744

It’s fun to laugh, but it’s a good reminder that there’s always going to be a weakness. You need to structure your business disaster recovery in a way that you can quickly and efficiently deal with a complete wipe-out like this.

And we’re done.

So, to summarise this essay, you need to:

Always install updates
Always take backups
Use AV software with full journaling and rollback capabilities
Practice sufficient paranoia

With these four steps, you will be as safe as the biggest corporate. (roughly speaking)

RingZer0team CTF - Challenge 44

Sat, 24 Jun 2017 00:00:00 +1200

Challenge 56 (“Hey Chuck where is the flag?”, under the Forensic Challenges) started by only offering a .pcap file. I downloaded Wireshark and had a quick dig. The packet capture consisted of a brief browse of a “Chuck Norris Facts” website.

After quickly flicking through the pcap itself looking for low-hanging fruit, I used File -> Export Objects -> HTTP. This takes all the HTTP traffic, recombines the packets into files, and saves them.

Once I’d done that, I opened up the files and had a quick flick through. I checked the images for metadata etc but didn’t find anything. However I did fine a picture of Chuck Norris, appearing to urge the reader to keep searching files – in French.

When I didn’t find anything super-obvious in the image contents, I hopped into Bash and just ran grep -i flag *. It quickly returned the flag, stored in a php file!

Nice and easy hunt, great way to finish the day.

RingZer0team CTF - Challenge 65

Sat, 24 Jun 2017 00:00:00 +1200

This is a continuation of my series on RingZer0Team.com.

Challenge 65 (“Hide my ass in my home”, under the Forensic challenges) is a nice quick little forensic challenge. You get to download a .tar file, and do a bit of digging to uncover the flag.

This was a nice easy few minutes. After downloading the archive, I discovered a few files inside - including a particularly groovy electronica swing track, which I’m listening to again right now. Yeaaaaahhhhh.

Apart from a groovy mp3, the archive contains some dot-files such as .bash_history, a .jpg picture, an index.html, and some other bits.

I found the Flag in one of the dot-files; specifically, it was inside a Vim swap file that seemed to be leftover from a power failure or similar.

WireGuard - Part Three (Troubleshooting)

Mon, 12 Jun 2017 00:00:00 +1200

This is part of my brief series on WireGuard. I’m pretty enamoured with WireGuard and the way it works, but there were a couple slightly curly bits that I needed to get my head around. This troubleshooting guide is a rough dump of the issues I had, and how I fixed them.

Gotten Stuck?

At this stage, there are actually a few ways that this can go wrong, even though we haven’t done much. Think through all the bits:

Installed WireGuard at both ends
Set up your NAT rule on the remote side
Created a private and public key on each side
Put each public key in the opposite side’s [Peer] config
Put a suitable IP address on each side

Troubleshooting

There are, actually, a myriad of ways this can fail:

Broken routing on the client PC
Broken NAT on the remote router
Broken routing on the remote PC
Broken wg0 configuration on one side
Lack of TCP forwarding on the remote computer
Lack of Proxy ARP on the remote computer

If you can’t ping the remote server yet, don’t panic. Run tcpdump to find out what you’re missing.

On the remote server: tcpdump -i wg0
On your local machine: ping -c1 10.20.40.1

That will tell you whether your packets are reaching the remote server, or if they’re not getting through the tunnel.

If they’re not making it through the tunnel at all, you’ll probably be seeing error messages in the ping. For example:

PING 10.20.10.1 (10.20.10.1) 56(84) bytes of data.
From 10.20.40.2 icmp_seq=1 Destination Host Unreachable
ping: sendmsg: Required key not available

That’s the error message I got when I set AllowedIPs too strictly. Because I was trying to ping something that was routable, but wasn’t within the AllowedIPs range, there was no applicable key for the packet.

Thanks

Huge gratitude to Jason Donenfeld (aka zx2c4) for spending his time not only reading this post, but also for sending me some fantastic feedback! I’d made some bungles in my original post on this topic, and he vastly helped my understanding.

Also, huge gratitude to another Jason (@rendition) who has helped me develop from a junior network admin into a … ‘moderate’ network admin. I’ve learned more in the last year than I ever thought possible. He’s taught me nearly everything I know about managed networking, reviewed this post for me, and is actually the guy who introduced me to WireGuard originally!

WireGuard - Part Two (VPN routing)

Sun, 11 Jun 2017 00:00:00 +1200

This is a continuation of my brief series on the new WireGuard VPN. Part One was about the simple building-blocks to get WireGuard working between two endpoints. Now that we’ve got a couple machines able to ping each other by IP address, we can carry on a bit deeper into the inter-LAN routing stuff.

Extending on from the IP addresses in Part One, instead of JUST connecting to the remote machine, I want to actually have access to everything on the whole 10.20.0.0/16 network; even the non-WireGuard devices. I want it to be like I’m there on-site. The idea here is roughly:

Remote machine: wg0: 10.20.40.1 (behind public IP 163.172.161.0)
Local machine: wg0: 10.20.40.2 (with eth0 on 192.168.88.207)
Machines on remote LAN: 10.20.0.0/16
Remote LAN Watchguard range: 10.20.40.0/16 – note this is within the remote 10.20.0.0/16 range

From my local machine, with minimal interaction, I want to be able to ping something like 10.20.10.30, and get a response.

Happily, now that the two endpoints are talking with each other, there’s really not much that has to happen to get things working the way I want.

Remote Config

Many Linux distros disable IPv4 packet forwarding by default. But in this case, we very much want this enabled. Edit your /etc/sysctl.conf file, and look for the line: net.ipv4.ip_forward, and set the value to 1. You may need to add this line manually, or uncomment it.
You’ll also need to enable ProxyARP. This is another kernel setting that’s usually disabled, so edit /etc/sysctl.conf, and add the following line: net.ipv4.conf.all.proxy_arp = 1.
Reload your kernel settings. A reboot will do the trick, or you could just run sudo sysctl -p /etc/sysctl.conf.

Starting WireGuard on Boot

There are a variety options to do this, and if you’re a seasoned sysadmin you probably already have a favourite way.

One option is to add post-up wg-quick up wg0 to the tail of your eth0 block (or appropriate interface) in /etc/network/interfaces.

A second option (for systemd users) is to simply run systemctl enable wg-quick@wg0. This will tell systemd to bring the wg0 interface up once a network connection is established.

I only do this on the remote machine, because I don’t want my local machine to be forever spinning up it’s WireGuard connection; however I always want the remote machine listening for my packets. That said, there’s probably no reason to avoid this; I just haven’t done it yet.

Local Config

Here’s the really cool bit. Since your remote machine is now set up to perform IPv4 forwaring and ProxyARP, the only thing you need to change on the local side is a single number. Or rather, a couple numbers.

In your /etc/WireGuard/wg0.conf file, just expand the network range of your interface to include the entire remote LAN:
```
 [Interface]
 ...
 Address = 10.20.40.2/16

 [Peer]
 ...
 AllowedIPs = 10.20.0.0/16
```
Now reload your WireGuard config, either by rebooting, or running sudo wg-quick down wg0 && sudo wg-quick up wg0.

That’s all! When you make the above changes, wg-quick will modify your routing table so that any IP within 10.20.x.y will be sent over the wg0 interface. The remote host will dutifully forward the packet out into the rest of the network, and ProxyARP will take care of the rest:

$ ping -c1 10.20.10.31
PING 10.20.10.31 (10.20.10.31) 56(84) bytes of data.
64 bytes from 10.20.10.31: icmp_seq=1 ttl=63 time=91.8 ms

Troubleshooting

Okay so by now there’s a really good chance that you’ve bumped into trouble. I’ve started on a basic WireGuard troubleshooting guide, but it only covers the issues that I bumped into. If you’re still stuck after reading through that, ask a question on the WireGuard Mailing List, or reach out via IRC (#WireGuard on Freenode) - this is all on the WireGuard website.

Wrap-up

That should be all! When you reboot the remote server (I specify reboot, because you want to test that your solution can survive an outage at the remote site), you should be able to ping other things in the remote LAN without any additional interaction. I still manually run wg-quick up wg0 on my local laptop, because I don’t want to be connected remotely all the time.

Thanks

If you do end up using WireGuard, go forth and donate! Seriously, at very least, send Jason the cost of a local cup of coffee or a beer for his efforts.

WireGuard - Part One (Installation)

Sat, 03 Jun 2017 00:00:00 +1200

WireGuard is the most excellent VPN stack around. It’s really fast, the concept of Cryptokey Routing is awesome, and I love the speed and simplicity benefits that come from opionionated cryptography. The protocol is so simple - expressed in a mere 4k lines of code - that it’s auditable by anyone.

But.

With my initial naive approach, I found myself using HTTPS, over ports forwarded over SSH tunnels, connected over WireGuard. Although it was straightforward to get WireGuard working between two endpoints, I ended up in nested-crypto hell.

So, this brief series is about making WireGuard work as a VPN onto a different site. We’ll start by making it work between two endpoints, and we’ll go forward from there. My end goal is to have access to all the resources on a remote site, just by running wg-quick up wg0.

The endpoints I’m going to set up are at:

Remote machine: wg0: 10.20.40.1 (behind public IP 163.172.161.0)
Local machine: wg0: 10.20.40.2

NAT Setup

Chances are, your remote endpoint is behind a firewall of some sort. Pick a high port, and configure your firewall to forward UDP packets on that port through to your remote WireGuard endpoint. You don’t need to do this on your local side, because reply traffic from the remote side will generally be handled by the stateful session part of your firewall.

Many routers and firewalls offer port address translation (also known as PAT) as part of port forwarding or NAT. This is when a packet hits the firewall on (say) port 57432, and the firewall puts it on the LAN to port 22, for example. You don’t want this. I think it’s possible with WireGuard, but it adds complexity without benefit.

Config - Remote Site

Install WireGuard

Generate your keys. The following will create a public key and a stub config in /etc/WireGuard/:

 cd /etc/WireGuard
 umask 077
 printf "[Interface]\nPrivateKey = " > wg0.conf
 wg genkey | tee -a wg0.conf | wg pubkey > publickey

Edit your config to match:

 [Interface]
 PrivateKey = WhAt3v3R= (this is the private key generated on this machine)
 ListenPort = 12345 (this is the UDP port you've forwarded from your firewall)
 Address = 10.20.40.1/24 (this will be the IP given to the wg0 interface)

 [Peer]
 PublicKey = (leave this blank for now; you'll paste in your local public key here soon)
 AllowedIPs = 10.20.40.0/24 (this is the range of WireGuard IP addresses that this Peer's key can be used from)

That should be all you need to configure on the remote side for now. Save your wg0.conf file, and bring the interface up:
```
 wg-quick up wg0
 ping -c1 10.20.40.1
```
You should see a bunch of actions performed by wg-quick, and a reply packet from your ping. Now onto the local side.

Config - Local Machine

Install WireGuard and generate your keys, as per the first two steps above.

Edit your configuration again:

 [Interface]
 PrivateKey = WhAt3v3R+PaRt-tw0= (this is the private key generated on this machine)
 ListenPort = 12345 (this is the UDP port again; I don't think they have to be the same)
 Address = 10.20.40.2/24 (note - different IP address, but in the same range)

 [Peer]
 PublicKey = ?????????? (copy the public key from the REMOTE server here)
 Endpoint = 163.172.161.0:12345 (enter the PUBLIC IP address of the remote site, plus the forwarded port)
 AllowedIPs = 10.20.40.0/24 (specifying that packets using this key must come from within 10.20.40.x)

Now that you’ve got a local public key, take a second to paste it into the Peer public key section on the remote server.
Once you’re finished you should be able to bring the interface up and ping it:
```
 wg-quick up wg0
 ping -c1 10.20.40.2
```
… and you should even be able to ping the remote server as well:
```
 ping -c1 10.20.40.1
```

Gotten Stuck?

At this stage, there are actually a few ways that this can go wrong, even though we haven’t done much. Here’s a quick summary of everything we’ve done:

Installed WireGuard at both ends
Set up your NAT rule on the remote side
Created a private and public key on each side
Put each public key in the opposite side’s [Peer] config
Put a suitable IP address on each side

If you’ve nailed each of those and you’re still having trouble, you can have a quick look at the brief Troubleshooting guide I’ve put together as Part Three.

Onto Part Two

That should be all you need to get WireGuard working between two machines on two different sites. So far we haven’t done anything either interesting or uncommon - this is all the basic stuff you’ll find on the WireGuard Quick Start page, although expressed slightly differently. The next article will be a bit more about intra-site routing.

Thanks

If you do end up using WireGuard, go forth and donate! Seriously, at very least, send Jason the cost of a local cup of coffee or a beer for his efforts.

RingZer0team CTF - Linux Sysadmin challenges

Sun, 28 May 2017 00:00:00 +1200

The Sysadmin Linux series of challenges is where you’re trying to breach the security of a Linux system. I actually finished most of these last year, but I wanted to finish my last two. Of course, to get to the last two stages, you need to use the flags from the previous stages. So I’m revisiting them.

Level 1 - Morpheus -> Trinity

We start by SSH’ing into a particular user account on the ringzer0team server:

You have mail.
Last login: Thu Apr 27 02:52:40 2017 from <somewhere>
morpheus@forensics:~$

There’s only one file in the home folder, and I can’t read it. There’s a /home/trinity folder with full read access, but also nothing legible.

The flag for level 1 is found by running ps aux, which reveals what appears to be Trinity’s password:

root 3241 0.0 0.0 4188 572 ? S Jan14 1:44 /bin/sh /root/backup.sh -u trinity -p Flag-<redacted>

Level 2 - Morpheus -> Architect

Aha, and the flag for level two is in /etc/fstab, which contains what appears to be The Architect’s password, in base64:

/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0
/dev/fd0 /media/floppy0 auto rw,user,noauto 0 0
#//TheMAtrix/phone /media/Matrix cifs username=architect,password=$(base64 -d "<redacted>"),iocharset=utf8,sec=ntlm 0 0

Level 3 - The Architect

Level 3 requires you to log on as architect, using the password we discovered earlier. The only clue is “dig for password”.

This bit got a bit harder. There are a couple possible hints in architect’s mail file, including a bunch of attempts to mount a cifs volume, and reference to a file in /backup/.

Digging through the files in /backup left me at a loss. There are references to /tmp/Gathering.py, but that’s a challenge for later (cypher’s challenge). After some digging I located /var/tmp/.swl, which contained some MySQL credentials!

mysql> show tables;
+----------------+
| Tables_in_arch |
+----------------+
| arch |
| flag |
+----------------+
2 rows in set (0.00 sec)

mysql> select * from arch;
+------+-----------------------+
| id | arch |
+------+-----------------------+
| 1 | The one |
| 1 | The null one |
| 1 | The mother of the one |
| 1 | The father of the one |
| 1 | The flag of the one |
| 1 | The null one |
+------+-----------------------+
6 rows in set (0.00 sec) 

mysql> select * from flag;
+---------------------------------+
| flag |
+---------------------------------+
| FLAG-<redacted> |
+---------------------------------+
1 row in set (0.00 sec)

Yasss after a solid hour of digging, I’ve found the flag for the third level!

Level Four - Morpheus -> Oracle

Again we’re logging on as morpheus, but this time the goal is to access the oracle account.

Lots of tasty stuff under /var/tmp/.viminfo, now that I’ve found it. References to files in /backup/, to /tmp/Gathering.py, /tmp/mail_cypher

Don’t make the mistake of trying to cat .swo; it’ll eat your console session.

Found this interesting thing in mail, but it’s not helpful: forensics.localdomain : Apr 14 21:14:33 : morpheus : user NOT in sudoers ; TTY=pts/4 ; PWD=/home/trinity ; USER=root ; COMMAND=/usr/sbin/usermod -aG neo morpheus

In one of the /backup files, I found the remains of an SSH key, and it worked! Access to the oracle account achieved. In the home folder, we’ve got a handy-dandy Base64-encoded flag in flag.txt.

Level Five - Oracle Encrypted File

There’s a file in the oracle home folder named encflag.txt.enc:

U2FsdGVkX1+dCl4WEHNJKBqA8a4fQeheOgA7oiNmjwlJQvGaQAgqcIsGRIcbdHKF
heSs51JRSEmOLqVyGvoxDA--

Decoding the base64 returns a binary blob, prefixed with the string “Salted”. This is one of the levels that I never completed originally, so maybe it’s time to give it another crack.

A quick google of “salted base64” reveals an interesting comment on StackExchange. Apparently I’m looking at a file encrypted with the OpenSSL ’enc’ command, so I first copy the file to my own computer to work.

I discover that the command openssl enc -in testfile -d -a will un-base64 it, then load the file in. I need to pass it a password to decrypt with though, so now I need to find that.

Oh my god I just did cat .* in the oracle home folder and literally found an alias that reads and decrypts the file. I can’t believe that took me almost an hour.

There are no words.

Level Six - Trinity -> Neo

Okay getting serious now. Using Trinity’s account, I need to find the password for the neo account.

It’s a bit noisy on login:

You have mail.
Last login: Tue Apr 25 16:29:29 2017 from <le blah>
-bash: hello: command not found
Sup Neo!
ls: cannot open directory /home/neo: Permission denied
cat: phonebook: Permission denied

A vague tickle in my memory prompted me to run sudo -l right off the bat. Rewarded with:

User trinity may run the following commands on this host:
 (neo) /bin/cat /home/trinity/*

So of course I immediately run sudo -u neo /bin/cat /home/trinity/*

This gives us a ‘phone book’ containing:

The Oracle 1800-133-7133
Persephone 345-555-1244





copy made by Cypher copy utility on /home/neo/phonebook

Trinity’s .bashrc contains a bit of kruft:

hello neo
echo "Sup Neo!"
$(ls -lart /home/neo)
cat phonebook

Really not useful. I remember finding Trinity’s password with ps aux:

root 3241 0.0 0.0 4188 572 ? S Jan14 1:47 /bin/sh /root/backup.sh -u trinity -p Flag-08grILsn3ekqhDK7cKBV6ka8B

It looks a little bit like a MySQL connection string, so I try that but no dice. Nothing under /var/tmp this time, and grep doesn’t find anything useful under the /backup goldmine.

There’s some weirdness under /etc/passwd. I don’t know what this means, but saving it for later:

trinity:x:1002:1002:trinity,%,lsdf(940998+(n.~,3):/home/trinity:/bin/bash

I looked through the mail file, but nothing jumped out at me there. Also trawled again through /backup, as well as /var/backups. Then /etc and /var. Argh.

Finally I start looking for Persephone. Nothing helpful, but it reminds me of the last line in the phonebook.

copy made by Cypher copy utility on /home/neo/phonebook

And then I remember that the sudo command allows a * at the end of the cat command, which might allow me to do path traversal…

sudo -u neo /bin/cat /home/trinity/../neo/phonebook

Finally, after a whole hour of digging, I’m awarded Neo’s password!

Level 7 - Neo is Not Alone

I don’t know what this clue means, but I bet it has something to do with Persephone. Persephone comes from Greek mythology as one of Zeus’ daughters, and the goddess of the underworld. Does that help me here? I don’t know, it’s getting late…

I run cat .* in Neo’s home directory to see if there’s anything hiding there, but no luck this time. Also nothing for sudo -l.

There’s an unreadable file (owned by root) called ‘result.stat’, that’s probably related to something. I browse the mail history and see:

forensics.localdomain : Dec 7 14:08:08 : neo : user NOT in sudoers ; TTY=pts/2 ; PWD=/home/neo ; USER=morpheus ; COMMAND=/home/morpheus/egrep /home/morpheus/../neo/result.stat

That could be handy. I do remember an egrep in /home/neo, will have a look. Lots of reference to /bin/monitor, too.

I’m only in the neo group, so nothing to see there.

Running ps aux | grep neo shows a heap of /bin/monitor processes started by root, but running under neo. There’s also a “SCREEN” instance there, but running screen -D -RR doesn’t reclaim it, so something’s up there. I’m pretty sure there’s something there in /bin/monitor.

In all honesty this one had me beat. It was half-past midnight and I was exhausted. I knew that there was something special about /bin/monitor, and I knew there was something special about the fact that it was running as neo. But I took the expedient route and had a quick google. I found the answer at http://blog.dornea.nu/2016/10/30/ringzer0-ctf-sysadmin-linux/ - without this I could have been looking for years. However, as is so often the case, this tutorial was the basis of some great learning for me.

Because the process is owned by neo, and because /bin/monitor could be read by neo, it meant I could run a stack trace. I have never done this before - no false impressions here, I totally relied on that dornea.ru page to tip me off to the usage of strace. But once I knew that, I had the flag.

I was very well misled by this one, actually. The Persephone comment in the phonebook totally led me down the wrong path, and I spent quite a long time searching for her details on the system. It was the reddest herring possible.

Level 8 - Morpheus -> Cypher

Okay it’s late now so I’m going to try to thrash this one quick. Nothing in ps aux for cypher. Grepping /backup found a python file and a dump showing a cron job runs this file every minute. The contents of the python file are:

import os
os.system('ps aux > /tmp/28JNvE05KBltE8S7o2xu')

I can’t read the output file, but I can edit the python script! This is gonna be easy.

After a little bit of trial and error, I edited the python script to first list the contents of the /home/cypher folder, and after that, to output the contents of /home/cypher/flag.txt

It’s important to note that the python file is recreated every three minutes when it’s run, so you need to redo your changes for every iteration.

Fin

That’s the end! I’m quite pleased I’ve managed to finish all of the Sysadmin Linux section. Of course that’s also scary, because it means I need to break into the other sections. That’s OK though… it’s a good time to do it!

RingZer0team CTF - Challenge 148

Sat, 27 May 2017 00:00:00 +1200

Challenge 148 (“Sysadmin Linux Level 2”) is one of a series of challenges where you’re trying to breach the security of a Linux system. I actually finished most of these last year, but I wanted to finish my last two. Of course, to get to the last two stages, you need to use the flags from the previous stages. So I’m revisiting them.

We start by SSH’ing into a particular user account on the ringzer0team server:

You have mail. Last login: Thu Apr 27 02:52:40 2017 from morpheus@forensics:~$

There’s only one file in the home folder, and I can’t read it. There’s a /home/trinity folder with full read access, but also nothing legible.

Running ps aux reveals what appears to be Trinity’s password:

root 3241 0.0 0.0 4188 572 ? S Jan14 1:44 /bin/sh /root/backup.sh -u trinity -p Flag-

Aha, and /etc/fstab contains what appears to be The Architect’s password, in base64:

/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0 /dev/fd0 /media/floppy0 auto rw,user,noauto 0 0 #//TheMAtrix/phone /media/Matrix cifs username=architect,password=$(base64 -d “”),iocharset=utf8,sec=ntlm 0 0

Great! Now I can carry on to my actual goal, which was to nail challenge 91 (“Sysadmin Linux Level 7”)!

RingZer0team CTF - Challenge 57

Fri, 05 May 2017 00:00:00 +1200

This is a continuation of my series on RingZer0Team.com.

Challenge 57 (“Hash Breaker Reloaded”, under the Coding Challenges) is one of a series of challenges where you’re simply presented with a hash - you need to return the plaintext value to the page within 3 seconds.

In contrast to Challenge 56, this challenge also includes a salt:

You have 3 seconds to break this hash
Send the answer back using https://ringzer0team.com/challenges/57/[clear_text]


----- BEGIN HASH -----
ab9507edbb2501b3c02e47c51af0178d68655980
----- END HASH -----

----- BEGIN SALT -----
c2ac9d8d004b4011d0864e76c7ebaaccfd18464bb8ff66bdbf19a703eb95a944
----- END SALT -----

The hash looks like another SHA-1, but of course the online hash reversers don’t have the now-salted hash in their rainbow tables. I figured that the challenge was a simple continuation of the previous challenge, so I crossed my fingers and guessed the hash was simply a number (from last time), concatenated with the plain-text salt.

… And I was lucky! The below code got me the flag:

#!/usr/bin/python3
from requests import get
from hashlib import sha1
from re import search

url='https://ringzer0team.com/challenges/57'
session=dict(PHPSESSID='<blahblah>')
resp=get(url,cookies=session)
hash=resp.text.split("-----")[2].split("\t")[2].split("<")[0]
salt=resp.text.split("-----")[6].split("\t")[2].split("<")[0]
newhash=''
salt=salt.encode('utf-8')
x=0
while newhash != hash:
		newhash=sha1(str(x).encode('utf-8')+salt).hexdigest()
		print("%s = %s" % (newhash, x))
		x+=1
	
resp=get(url+'/'+str(x-1),cookies=session)
flag=search("FLAG-.{24}",resp.text).group()
print(flag)

As last time, there’s no error handling in that code, so if it runs for more than a few seconds it’s probably toast.

RingZer0team CTF - Challenge 56

Tue, 02 May 2017 00:00:00 +1200

Challenge 56 (“Hash Breaker”, under the Coding Challenges) is one of a series of challenges where you’re simply presented with a hash - you need to return the plaintext value to the page within 3 seconds.

Of course, this is impossible to do manually, so it’s a programming challenge.

The hash looks like a simple SHA-1, so I put it into an online hash reverser and discovered it’s just a SHA1 hash of a random number. SHA1 is really fast, so this should be simple.

Here’s how I tackled the problem:

#!python
# It's in Python3, because of reasons
from requests import get
from hashlib import sha1
from re import search
calculated_hash=''
x=0

# Your session cookie needs to go here
session=dict(PHPSESSID='<cookie contents>')

# Load the web page
url='https://ringzer0team.com/challenges/56'
resp=get(url,cookies=session)

# Extract the hash from the page contents
target_hash=resp.text.split("-----")[2].split("\t")[2].split("<")[0]

# Iterate hashes from zero until the calculated hash matches the target hash
while calculated_hash != target_hash:
		calculated_hash=sha1(str(x).encode('utf-8')).hexdigest()
		print("%s = %s" % (calculated_hash, x))
		x+=1

# If we got here, yay! We found the key. Now submit it back to the site, and print our magic flag
resp=get(url+'/'+str(x-1),cookies=session)
flag=search("FLAG-.{24}",resp.text).group()
print(flag)

There’s no error handling in that code, so if it runs for more than a few seconds it’s probably toast.

HP Procurve's warn-and-disable

Sun, 30 Apr 2017 00:00:00 +1200

Since mid-2016, I’ve been working as a Network and Security Administrator. While I’d done a fair amount of networking previously, most of my experience had been with either unmanaged switches, or in a pre-built Cisco environment. Stepping into the world of managed networking was new for me, as was stepping into the world of HP Switches.

We were having recurring issues with a certain business unit looping ports on a switch. We had loop-protect running, but it was only set to disable the port after 5 seconds, and only for 300 seconds. This wasn’t long enough (we’ve since set it to something more resilient).

During our attempts to minimise the impact of a network loop, we enabled this setting on our HP Procurve switches:

fault-finder broadcast-storm action warn-and-disable

This seems simple enough. The fault-finder module will look for broadcast storms, and when found, will throw a syslog, an SNMP trap, and will disable the offending switch port.

Don’t be fooled. This will absolutely wreck your breakfast.

The fault-finder module is a system-wide setting, which means it applies to uplink ports as well. When a broadcast storm happens, all your distribution switches will see the broadcasts coming from… their uplink port. So of course, the switches disable their uplink to the rest of the network, neatly segregating themselves away from any services or central management.

Suffice to say, the next time we had a network loop there was a lot of walking and console cable madness, while we visited each of the distribution switches and manually re-enabled the disabled ports.

Lesson learned, loop-protect it is. (Until we get STP up campus-wide, of course)

Making Lektor work with grsecurity

Sat, 29 Oct 2016 00:00:00 +1300

I started using grsecurity on my servers in 2015, and there’s always a bit of tuning required.

I was recently playing with Lektor (before I swapped to Pelican), and I had a bit of trouble with my grsec kernel. In particular, Lektor and Pelican are both run within a virtualenv Python environment, and grsec eats it like popcorn in two different ways:

TPE (Trusted Path Execution) throws a wobbly:

[253241.370019] grsec: From {ssh-origin-ip}: denied untrusted exec (due to file in world-writable directory) of /tmp/#50 by /usr/local/lib/lektor/bin/lektor[lektor:60593] uid/euid:1000/1000 gid/egid:1000/1000, parent /bin/bash[bash:60581] uid/euid:1000/1000 gid/egid:1000/1000

This block occurs because the virtualenv violates grsecurity’s Trusted Path Execution protection - the Python executable isn’t under a trusted path, and the user isn’t in the TPE-bypass group.

To resolve this one, add your user to the TPE group. This will allow the user to execute binaries which aren’t in trusted locations:

sudo usermod -aG grsec-tpe $USER

RWX protection. This is a common problem with Python-based apps, even though Python has an exception (Edit 2020-04-12 - Issue 6 from https://github.com/thestinger/paxd/ has been deleted) by default.

[ 207.534876] grsec: From {ssh-source-ip}: denied RWX mmap of <anonymous mapping> by /usr/local/lib/lektor/bin/lektor[lektor:534] uid/euid:1000/1000 gid/egid:1000/1000, parent /bin/bash[bash:477] uid/euid:1000/1000 gid/egid:1000/1000

First, create your PaX ELF headers, and then disable grsec’s MEMPROTECT extension on the included python2 binary, while you enable EMUTRAMP:

sudo paxctl /usr/local/lib/lektor/bin/python2 -c
sudo paxctl /usr/local/lib/lektor/bin/python2 -Em

Result!

Broken log times in APC's PCNS Appliance 4.1

Thu, 27 Oct 2016 00:00:00 +1300

Schneider Electric’s PowerChute Network Shutdown is a piece of software which communicates with your local UPS, and initiates system shutdown if the UPS battery is unable to continue providing power. This helps to preserve file integrity in the event of a prolonged power failure.

Previously, you had to install the PCNS client separately on each virtual machine. Since then though, APC have released a PCNS VMware Appliance which is installed directly into vCenter, and initiates shutdown on all the VMware guests through a single Virtual Machine. This is a much tidier model, so we’ve recently embarked on migrating to the PCNS Appliance.

After installing the PCNS 4.1 appliance and getting everything working, I discovered that the time on all the log files was wrong by a significant margin. I can’t remember exactly what the margin was, it could have been say 8 hours or something.

I did a bit of research and eventually I discovered a spectaularly old question on the APC forums, dating all the way back to 2011:

“I have installed PCNS 2.2.4 into the vMA of an ESXi 4.1 server. It properly communicates with the associated NMC and performs correctly. However, it reports all the activities in Eastern Standard Time. I don’t know where it’s picking up that time zone and can’t find anywhere to change it. In the vSphere client the time shows up correctly. Checking date inside the vMA from the CLI also shows the correct time and timezone. The date and time setting in the NMC is correct and is set to synch with NTP with correct time zone.”
From: http://forums.apc.com/spaces/7/ups-management-devices-powerchute-software/forums/general/6809/pcns-2-2-4-esxi-annoyance

The answers include a broken link to an APC web page about it, but fortunately the poster copied the original post detail, which showed me where to look:

“PCNS uses a standard java function to retrieve the current date from the system when writing events to the log. The Problem on the VIMA is that this function always returns the US date format (PDT).
This is an issue with Java http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6456628
You can correct the time issue on VIMA by modifying /etc/sysconfig/clock file.
+/etc/sysconfig/clock contains a line ZONE=“America/Los_Angeles”+
You need to edit the ZONE= to match your region such as ZONE=“America/New_York”

Sounds great! But when I looked in /etc/sysconfig/clock, I discovered my time zone was already set to “Pacific/Auckland”.

However, it also contained the value “UTC = False”. I changed this to “UTC = True”, and et voilà! The problem is fixed!

Using ECDSA certificates with Let's Encrypt

Thu, 27 Oct 2016 00:00:00 +1300

Let’s Encrypt’s Certbot will generate an RSA key by default. But we want to step into the new and exciting world of elliptic curve cryptography! Unfortunately Certbot doesn’t really roll that way, so there are a couple hoops to jump through first.

A word of caution: this post has been written in retrospect, some time after I actually got ECDSA working. That means there are bound to be squiggly little steps that I’ve missed, and I certainly should have provided screenshots or snippets that I’ve missed. Sorry.

This post assumes you’ve already installed Certbot. I had a working regular certificate from Certbot before I changed to ECDSA, so if you have problems following this from scratch, I do recommend trying that first.

For future reference, I’m running the latest certbot available in Debian unstable, which is version 0.8.1-3.

I got most of this information from Scott Helme’s website, which has been awesome.

In brief:

Generate yourself an ECDSA private key:

openssl ecparam -genkey -name secp384r1 | openssl ec -out ec.key

You can change the curve that you use, if you feel a bit wiggly about the controversy around the NSA & NIST degrading the quality of the curves. I don’t feel particularly wiggly about that, myself.

Generate a Certificate Signing Request (CSR) with your shiny new key:

openssl req -new -sha256 -key ec.key -nodes -out ec.csr -outform pem

That will give you ec.key (your private key), and ec.csr (your certificate signing request). Time to get Let’s Encrypt involved.

Create your certificate:

certbot certonly -w /var/www/html/ -d {your_domain} –email “{your_email}” –csr ./ec.csr –agree-tos

If everything goes perfectly, that should leave you with a new shiny set of certificates – quite possibly named something clumsy like 0000-cert.pem and 0001-fullchain.pem, or similar. Throw those into your nginx config and give it a test to see if it’s working.

Schedule your certificate renewals:

I had particular trouble with the renewal process of ECDSA certificates, because certbot renew isn’t compatible with custom CSR’s. You need to run certbot certonly to pass the –csr argument, and then you need to deal with the output yourself.

Even more irksome, the certonly function will fail if you ask it to renew certificates which already exist:

An unexpected error occurred:
OSError: [Errno 17] File exists: '/etc/letsencrypt/live/{your_domain}/cert_ecdsa.pem'
Please see the logfiles in /var/log/letsencrypt for more details.

There doesn’t seem to be any way to tell certbot to overwrite the old certificates automatically, so I created a /etc/letsencrypt/temp folder, and wrote up a really yuck cron job for it. If I were working on a production system I’d do something better, but this works for my lowly domain:

# Recreate certs under /etc/letsencrypt/temp
30 2 24 * * certbot certonly -w /var/www/html/ -d {your_domain} --email "{your_email}" --csr /path/to/your/ec.csr --agree-tos --non-interactive --webroot --cert-path /etc/letsencrypt/temp/cert_ecdsa.pem --fullchain-path /etc/letsencrypt/temp/fullchain_ecdsa.pem

# Backup current certs to home dir
31 2 24 * * mv /etc/letsencrypt/live/{your_domain}/cert_ecdsa.pem ~/certbackup/
31 2 24 * * mv /etc/letsencrypt/live/{your_domain}/fullchain_ecdsa.pem ~/certbackup/

# Move new certs to live folder
32 2 24 * * mv /etc/letsencrypt/temp/* /etc/letsencrypt/live/{your_domain}/

# Restart nginx
33 2 24 * * service nginx restart

And that’s all! It seems to be working so far, but I’m sure something is going to fail at some point – maybe I’ll hit my request limit, or the Let’s Encrypt service will be down, and I’ll lose my certs. If it happens too often I’ll come up with a nicer cronjob and update this post.