Linux on Such geek. Wow.

DR planning (aka how to rebuild my PC)

Thu, 27 Aug 2026 00:00:00 +1200

This one is written for future me, when I inevitably suffer the consequences of buying a $150 micro-PC from AliExpress equipped with an SSD that was swept off the floor after a flood.

… I regret nothing.

Anyway.

The blog itself is just static files (at some point I’ll write up the Pelican -> Hugo journey), so although the blog will keep running, I’ll have lost the ability to change anything. This blog has been written as I’ve set up my toolchain on a brand-new machine, so it’s kinda my disaster recovery plan.

The toolchain

I run Debian, and this blog is based on Hugo now, so grab the extended Hugo binary from the releases page and get the non-snapd version of Dart Sass.

sudo apt install -y git curl golang-go
tar xzf ~/Downloads/hugo_extended_*.tar.gz -C ~ hugo
tar xzf ~/Downloads/dart-sass-*-linux-x64.tar.gz -C ~/Downloads
sudo mv ~/hugo /usr/local/bin/
sudo mv ~/Downloads/dart-sass /opt/
sudo ln -sf /opt/dart-sass/sass /usr/local/bin/sass
hugo version

The keys

New machine, new keys, or so they say.

Lol nevermind, nobody says that. Restore them from my password manager like an adult. Then rebuild ~/.ssh/config. (make sure the blog server is the top line, for reasons that will soon become apparent)

Restore the blog content

git clone git@github.com:rhyven/ericlight.com.git ericlight.com
cd ericlight.com
hugo mod get
hugo --minify --gc

That’ll bring down the content of the blog itself, along with go.mod which will automatically pull down the Stack theme once we run hugo mod get. This is the point where I can have a look with hugo server and have a sense-check.

Incidentally, that also pins the Stack theme to the version I was running; update that with:

hugo mod get -u github.com/CaiJimmy/hugo-theme-stack/v4
hugo mod tidy
rm -rf public resources/_gen
hugo server

[this will absolutely rumble CSP ffs]

The publish script

Recreate publish_blog.sh:

#!/usr/bin/env bash
set -e
SERVER=$(awk '/^Host /{print $2; exit}' ~/.ssh/config)
cd ~/ericlight.com
rm -rf public resources/_gen
hugo --minify --gc
rsync -az --delete --dry-run --itemize-changes public/ "$SERVER:/var/www/ericlight.com/"
echo
read -r -p "Dry-run info above. Hit Ctrl+C now if that looks cursed, or Enter to publish."
rsync -az --delete public/ "$SERVER:/var/www/ericlight.com/"

Oh yeah, it’s all coming together.

Writing a post

Hugo posts are page bundles (unlike Pelican, from which I’ve recently migrated), so it’s a directory with an index.md in it, and the images live alongside:

mkdir ~/ericlight.com/content/post/yeah-so-my-ssd-finally-shat-the-bed

Front matter needs title, date, categories, tags, summary, and maybe even a hero image to make it all look nice.

Images go in the post’s own folder, referenced by bare filename:

![Alt-text for screen readers](screenshot.png "The caption everyone sees")

Leave the caption off and the theme just shows the alt text instead, which is usually fine.

draft: true plus hugo server -D to preview before committing to anything.

Publishing it

git add -A
git commit -m "New post: ssd failure post, as predicted"
git push
./publish_blog.sh

git add -A rather than git add ., because -A catches deletions too.

What about things that aren’t the blog, Eric?

There isn’t much! I’d need to reinstall Steam and StarCraft, reinstall the ESPHome builder and connect it to Home Assistan… that’s about it.

In fact, this article’s mere existence illustrates a regression towards infrastructure managment that I’m not entirely pleased with…

Until next time!

task pve-bridge blocked for more than 120 seconds - on Proxmox with MDATP

Sun, 13 Mar 2022 00:00:00 +1300

Today I updated and rebooted a bunch of servers - all in a day’s work. Everything went fine, until one of the hosts (the big one) didn’t boot up again properly.

I’ve got two hosts, both running ProxmoxVE (version 7.1) and Microsoft Defender for Endpoint (version 101.61.69-insiderfast). On reboot, one of the hosts came up fine; the other didn’t.

Long story short, uninstalling mdatp fixed the problem - wild hypothesis, but I suspect it was interfering with either network bridge or interprocess communication.

The rest of this post is technical info for people who might be able to fix it, and so that this blog post will come up for the other people who inevitably bump into this!

Symptoms - upon boot, the host came up, was apparently able to launch one VM, and recieved a timeout for all subsequent VM’s:

The lines are truncated, but all those pink lines are showing TASK ERROR: start failed: command '/usr/bin/kvm -id 153 [...]' failed: got timeout.

I say ‘apparently’ launched one VM, because I was unable to interact with it - in fact all the launch-on-boot VM’s appeared to have booted, except everything about them was non-responsive, including Proxmox stats windows and console.

Trying to manually interact with the VM’s using qm failed entirely - the console froze, completely unresponsive to either ^C or ^X, until I forcefully disconnected from SSH.

Checking dmesg revealed a horror story:

Mar 13 16:54:36 buckwheat kernel: [ 159.853700] vmbr1: port 1(eth3) entered blocking state
Mar 13 16:54:36 buckwheat kernel: [ 159.853703] vmbr1: port 1(eth3) entered forwarding state
Mar 13 16:54:43 buckwheat kernel: [ 166.700586] ixgbe 0000:81:00.1 eth3: NIC Link is Down
Mar 13 16:54:43 buckwheat kernel: [ 166.701135] vmbr1: port 1(eth3) entered disabled state
Mar 13 16:54:45 buckwheat kernel: [ 168.294208] ixgbe 0000:81:00.1 eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX
Mar 13 16:54:45 buckwheat kernel: [ 168.294343] vmbr1: port 1(eth3) entered blocking state
Mar 13 16:54:45 buckwheat kernel: [ 168.294349] vmbr1: port 1(eth3) entered forwarding state
Mar 13 16:54:47 buckwheat pvestatd[5040]: VM 112 qmp command failed - VM 112 qmp command 'query-proxmox-support' failed - unable to connect to VM 112 qmp socket - timeout a>
Mar 13 16:54:50 buckwheat pvestatd[5040]: VM 105 qmp command failed - VM 105 qmp command 'query-proxmox-support' failed - unable to connect to VM 105 qmp socket - timeout a>
Mar 13 16:54:53 buckwheat pvestatd[5040]: VM 115 qmp command failed - VM 115 qmp command 'query-proxmox-support' failed - unable to connect to VM 115 qmp socket - timeout a>
Mar 13 16:54:55 buckwheat pve-guests[11257]: start failed: command '/usr/bin/kvm -id 153 -name Eden -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/153.qmp,s>
Mar 13 16:54:56 buckwheat pvesh[5530]: Starting VM 153 failed: start failed: command '/usr/bin/kvm -id 153 -name Eden -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qem>
Mar 13 16:54:56 buckwheat pve-guests[5530]: <root@pam> end task UPID:buckwheat:000015C7:00000710:622D6A6F:startall::root@pam: OK
Mar 13 16:54:56 buckwheat systemd[1]: Finished PVE guests.
Mar 13 16:54:56 buckwheat systemd[1]: Reached target Multi-User System.
Mar 13 16:54:56 buckwheat systemd[1]: Reached target Graphical Interface.
Mar 13 16:54:56 buckwheat systemd[1]: Starting Update UTMP about System Runlevel Changes...
Mar 13 16:54:56 buckwheat systemd[1]: systemd-update-utmp-runlevel.service: Succeeded.
Mar 13 16:54:56 buckwheat systemd[1]: Finished Update UTMP about System Runlevel Changes.
Mar 13 16:54:56 buckwheat systemd[1]: Startup finished in 4.603s (kernel) + 2min 54.823s (userspace) = 2min 59.427s.
Mar 13 16:54:56 buckwheat pvestatd[5040]: VM 104 qmp command failed - VM 104 qmp command 'query-proxmox-support' failed - unable to connect to VM 104 qmp socket - timeout a>
Mar 13 16:54:59 buckwheat pvestatd[5040]: VM 153 qmp command failed - VM 153 qmp command 'query-proxmox-support' failed - got timeout
Mar 13 16:54:59 buckwheat pvestatd[5040]: status update time (30.177 seconds)
Mar 13 16:55:18 buckwheat pvestatd[5040]: VM 104 qmp command failed - VM 104 qmp command 'query-proxmox-support' failed - unable to connect to VM 104 qmp socket - timeout a>
Mar 13 16:55:21 buckwheat pvestatd[5040]: VM 153 qmp command failed - VM 153 qmp command 'query-proxmox-support' failed - unable to connect to VM 153 qmp socket - timeout a>
Mar 13 16:55:24 buckwheat pvestatd[5040]: VM 112 qmp command failed - VM 112 qmp command 'query-proxmox-support' failed - unable to connect to VM 112 qmp socket - timeout a>
Mar 13 16:55:27 buckwheat pvestatd[5040]: VM 105 qmp command failed - VM 105 qmp command 'query-proxmox-support' failed - unable to connect to VM 105 qmp socket - timeout a>
Mar 13 16:55:30 buckwheat pvestatd[5040]: VM 115 qmp command failed - VM 115 qmp command 'query-proxmox-support' failed - unable to connect to VM 115 qmp socket - timeout a>
Mar 13 16:55:30 buckwheat pvestatd[5040]: status update time (30.186 seconds)
Mar 13 16:55:48 buckwheat pvestatd[5040]: VM 153 qmp command failed - VM 153 qmp command 'query-proxmox-support' failed - unable to connect to VM 153 qmp socket - timeout a>
Mar 13 16:55:51 buckwheat pvestatd[5040]: VM 104 qmp command failed - VM 104 qmp command 'query-proxmox-support' failed - unable to connect to VM 104 qmp socket - timeout a>
Mar 13 16:55:54 buckwheat pvestatd[5040]: VM 115 qmp command failed - VM 115 qmp command 'query-proxmox-support' failed - unable to connect to VM 115 qmp socket - timeout a>
Mar 13 16:55:57 buckwheat pvestatd[5040]: VM 105 qmp command failed - VM 105 qmp command 'query-proxmox-support' failed - unable to connect to VM 105 qmp socket - timeout a>
Mar 13 16:56:00 buckwheat pvestatd[5040]: VM 112 qmp command failed - VM 112 qmp command 'query-proxmox-support' failed - unable to connect to VM 112 qmp socket - timeout a>
Mar 13 16:56:00 buckwheat pvestatd[5040]: status update time (30.191 seconds)
Mar 13 16:56:00 buckwheat kernel: [ 243.301563] INFO: task wdavdaemon:5583 blocked for more than 120 seconds.

The wdavdaemon is shown blocking here, but pve-bridge turned up just as frequently in the logs:

root@buckwheat:/var/log# grep blocked syslog
Mar 13 18:34:14 buckwheat kernel: [ 242.802463] INFO: task wdavdaemon:5740 blocked for more than 120 seconds.
Mar 13 18:34:14 buckwheat kernel: [ 242.802658] INFO: task wdavdaemon:4899 blocked for more than 120 seconds.
Mar 13 18:34:14 buckwheat kernel: [ 242.802840] INFO: task pve-bridge:6117 blocked for more than 120 seconds.
Mar 13 18:34:14 buckwheat kernel: [ 242.802945] INFO: task pve-bridge:7734 blocked for more than 120 seconds.
Mar 13 18:34:14 buckwheat kernel: [ 242.803057] INFO: task pve-bridge:9120 blocked for more than 120 seconds.
Mar 13 18:34:14 buckwheat kernel: [ 242.803170] INFO: task pve-bridge:10283 blocked for more than 120 seconds.
Mar 13 18:36:15 buckwheat kernel: [ 363.631430] INFO: task wdavdaemon:5740 blocked for more than 241 seconds.
Mar 13 18:36:15 buckwheat kernel: [ 363.631610] INFO: task wdavdaemon:4899 blocked for more than 241 seconds.
Mar 13 18:36:15 buckwheat kernel: [ 363.631776] INFO: task pve-bridge:6117 blocked for more than 241 seconds.
Mar 13 18:36:15 buckwheat kernel: [ 363.631872] INFO: task pve-bridge:7734 blocked for more than 241 seconds.

So the interesting thing is that another node - foxtail - was also updated to the same version of both Proxmox and mdatp. The only obvious difference I can see are the numbers of VLANs each of them are bridging to:

This is a list of the bridges now, when they’re both working (sorry, Markdown hates tabs; the below should be four tidy columns):

root@foxtail:/var/log# brctl show
bridge name	bridge id		STP enabled	interfaces
fwbr101i0	8000.9e7164f22bab	no		fwln101i0
							tap101i0
vmbr0		8000.c2602668d5f4	no		eno1
							fwpr101p0
							tap100i0
							tap109i0

versus

root@buckwheat:/var/log# brctl show
bridge name	bridge id		STP enabled	interfaces
fwbr102i0	8000.6	abd66965438	no		fwln102i0
							tap102i0
fwbr104i0	8000.f60b3f1a3118	no		fwln104i0
							tap104i0
fwbr105i0	8000.961281d6d5dc	no		fwln105i0
							tap105i0
vmbr0		8000.90e2bad572a8	no		eth2
							tap112i0
							tap115i0
							tap153i0
vmbr0v5		8000.0ea35ee013ba	no		eth2.5
							fwpr104p0
vmbr0v666	8000.1a23193ccf17	no		eth2.666
							fwpr102p0
							fwpr105p0
vmbr1		8000.90e2bad572a9	no		eth3

But when buckwheat was broken, the majority of the bridges were not visible:

At some point during troubleshooting, I restarted pvedaemon - although the service had started correctly, the restart failed with error: timeout waiting on systemd.

Acronis Cyber Protect and Microsoft Defender on the same Linux system

Sun, 07 Nov 2021 00:00:00 +1300

A few months ago, I published what I learned from playing with the Linux version of Microsoft Defender for Endpoint a few months back. If you’re here, you’ll have possibly already read my original post. There was a bit of tuning required to get it to behave nicely; just adding some sensible audit logging exclusions.

Well the Acronis Cyber Protect backup platform has recently been added to the bundle, and it had a few specific requirements, so I’ve documented them here.

There are four Acronis processes which MDATP will spend a lot of time scanning and rescanning. Exclude those from the scanning engine by executing the following commands as root:

mdatp exclusion process add --path /usr/lib/Acronis/Schedule/schedul2-bin
mdatp exclusion process add --path /usr/lib/Acronis/APL/active-protection
mdatp exclusion process add --path /opt/acronis/aakore
mdatp exclusion process add --path /usr/lib/Acronis/BackupAndRecovery/mms

There are also five processes which need to be excluded from auditctl, unless you want your audit logs rotated every ten minutes. Again, these must all be run as root.

echo -a never,exit -S 41 -S 42 -F comm="mms" -k exclude_acronis >> /etc/audit/rules.d/01-exclusion.rules
echo -a never,exit -S 41 -S 42 -S 43 -F comm="adp-agent" -k exclude_acronis >> /etc/audit/rules.d/01-exclusion.rules
echo -a never,exit -S 41 -S 42 -S 43 -S 288 -F exe=/usr/lib/Acronis/APL/active-protection -k exclude_acronis >> /etc/audit/rules.d/01-exclusion.rules
echo -a never,exit -S 41 -S 42 -S 43 -F exe=/opt/acronis/bin/updater -k exclude_acronis >> /etc/audit/rules.d/01-exclusion.rules
echo -a never,exit -S 41 -S 42 -S 288 -F exe=/opt/acronis/aakore -k exclude_acronis >> /etc/audit/rules.d/01-exclusion.rules

Once you’ve created the exclusions, you can either reboot or run service auditd restart. Either way, make sure you run service auditd status to ensure nothing went wrong with the exclusions.

Microsoft Defender for Endpoint (mdatp) on Debian Sid

Wed, 07 Jul 2021 00:00:00 +1200

** 2021-08-01 Update: **

Linux doesn’t have many great antivirus options available.

Don’t get me wrong, there are actually a few options nowadays. We’ve got the ever-present ClamAV; BitDefender has a good reputation, but I haven’t played with it yet; Sophos AV for Linux seems to be in limbo. ESET for Linux exists, as does Symantec Endpoint Protection for Linux. That’s a pretty decent range - but they each have some real drawback. Assuming the price is right, you still see compatibility, usability, licensing dramas, or resource drain challenges.

Most recently, the Linux version of Microsoft Defender for Endpoint entered the playing field. I finally got to play with this and I gotta say…

Holy shit.

Look, I know we enjoy ragging on Microsoft. They’ve demonstrated some awful behaviour over the years, and every action still harkens back to the “Embrace, Extend, Extinguish” modus operandi. But they’re doing really great work with the Defender/DATP/MDFE/whatever line. Even the basic Defender has been steading climbing the ranks, to the point where virtually all guidance I’ve heard for the last two years is: “if you’re not going full EDR, just use regulr Defender”. It’s no SentinelOne or CrowdStrike, but overall Defender just Gets The Job Done.

So, I was excited to try the Linux version. Long story short, it was a dream. The installation process was streaks ahead of the competition. Resource usage is generally negligible, adding an average load of around 0.5% CPU, going up to about 16% CPU during a scan. There’s a robust health and connectivity test built-in. Basically the whole thing just works, to the same level we’ve learned to expect from the Windows version. Oh and this one has actual documentation which seems to be considered optional by some of the competitors. 🙃

However.

You may find that mdatp is frantically scanning files that freqently Do Things. Bind9 (named) for example, might be handling hundreds of queries a minute. Each one of them triggers not only an mdatp scan, but also auditd entries… which often logs more than once per query. I was finding logs being filled and rotated every seven minutes in some cases… and then punctuated by auditd yelling “my buffer is full, I can’t log all these events!”.

So as a result, I’ve spent quite a few hours learning how to wrangle auditd around exclusions. Here’s what I’ve learned:

You need to lie to install MDFE on Sid

Edit /etc/os-release and change VERSION_ID="11" to VERSION_ID="10". After that, you can install MDFE following the regular instructions. Don’t forget to change it back later!

Troubleshooting performance issues:

First step - find the processes that mdatp is spending the most CPU time on:

# Download Microsoft's parser (only do this once)
wget -c https://raw.githubusercontent.com/microsoft/mdatp-xplat/master/linux/diagnostic/high_cpu_parser.py

# Ask Defender to output the stats, and then parse it looking for high CPU:
$ mdatp diagnostic real-time-protection-statistics --output json | python high_cpu_parser.py
925 	mongodb		29184
37575	pulseaudio	47
21355	packagekitd	30
37588	dbus-daemon	26
37550	sshd		25

… Yikes, it’s probably not necessary to scan MongoDB thirty thousand times! You can exclude a process from mdatp like this: sudo mdatp exclusion process add --path /usr/bin/mongod

But that’s not all… even though mdatp is no longer scanning the application, you still have auditd frantically logging every single thing it does. That’s also going to cause periodic slowdowns as the auditd buffer fills up, overflows, panicks, and then purges the stuff it couldn’t write down.

Syslog/audit logging issues

You can find out if anything is spewing a billion entries into your audit log, by just grepping for the rotation in syslog:

# grep 'Audit daemon rotating' /var/log/syslog
Jul 6 01:00:08 spam auditd[28996]: Audit daemon rotating log files
Jul 6 04:00:05 spam auditd[28996]: Audit daemon rotating log files
Jul 6 06:57:01 spam auditd[28996]: Audit daemon rotating log files
Jul 6 09:57:18 spam auditd[28996]: Audit daemon rotating log files

… Filling up a log file every three hours is weird, but previously this server was doing it every few minutes. You can see what’s causing all the log rotations with:

# cat /var/log/audit/audit.log* | cut -d ' ' -f26 | sort | uniq -c
 138581 
 1 exe="/bin/bash"
 4 exe="/bin/hostname"
 2 exe="/bin/nano"
 684 exe="/lib/systemd/systemd"
 4754 exe="/lib/systemd/systemd-logind"
 209 exe="/lib/systemd/systemd-user-runtime-dir"
 119 exe="/opt/microsoft/mdatp/sbin/osqueryi"
 9 exe="/usr/bin/apt-get"
 2133 exe="/usr/bin/dbus-daemon"
 13212 exe="/usr/bin/perl"
 10784 exe="/usr/bin/pmxcfs"
 14 exe="/usr/bin/rrdcached"
 2 exe="/usr/bin/sort"
 4 exe="/usr/lib/postfix/sbin/cleanup"
 880 exe="/usr/lib/postfix/sbin/pickup"
 175 exe="/usr/lib/postfix/sbin/qmgr"
 27 exe="/usr/lib/postfix/sbin/smtp"
 4 exe="/usr/lib/postfix/sbin/trivial-rewrite"
 16 exe="/usr/sbin/cron"
 10326 exe="/usr/sbin/ebtables-legacy"
 10326 exe="/usr/sbin/ebtables-legacy-restore"
 3 exe="/usr/sbin/postdrop"
 5 exe="/usr/sbin/qmeventd"
 2 exe="/usr/sbin/sendmail"
 56 exe="/usr/sbin/smartd"
 3255 exe="/usr/sbin/sshd"

The number on the left shows the number of times that binary has had it’s activity logged. This should clearly show the problem executables… today we’re looking at perl, pmxcfs, and ebtables-legacy. Once you’ve got that, you need to determine which syscalls are part of regular noise:

# grep /usr/bin/pmxcfs /var/log/audit/audit.log* | cut -d ' ' -f 4 | sort | uniq -c
 10856 
 8142 syscall=263
 1357 syscall=43
 1357 syscall=84

Referring to this document, we can see these syscalls are sys_accept, sys_rmdir, and sys_unlinkat. Since pmxcfs is the ProxmoxVS Cluster File System, these are frankly all pretty normal syscalls for it to make. Let’s ignore them.

echo -a never,exit -S 43 -S 84 -S 263 -F exe=/usr/bin/pmxcfs -k exclude_PVE_internals >> /etc/audit/rules.d/01-exclusion.rules

^ Gotta do that as root, btw, or just append the line however you see fit. Once you’ve added exclusions for all of your high-noise entires, run service auditd restart and then service auditd status to make sure it worked properly.

On my Zentyal box I had to add a bunch of entries to exclude logging of certain high-noise things. For example, DNS query resolution (via named) and Samba activity:

cat /etc/audit/rules.d/01-exclusion.rules

-a never,exit -S 41 -S 42 -S 49 -S 82 -S 288 -F exe=/usr/sbin/named -k exclude_DNS_queries
-a never,exit -S 43 -F exe=/usr/sbin/winbindd -k exclude_Samba
-a never,exit -S 41 -S 42 -S 43 -S 87 -F exe=/usr/sbin/smbd -k exclude_Samba
-a never,exit -S 41 -S 43 -S 87 -F exe=/usr/sbin/samba -k exclude_Samba

A word of warning: Don’t just ignore everything willy-nilly. The audit logging system exists so you can track activity on the system - blithely sending 100% of the log entries to the bin will undermine what the audit sytem exists for!

… That’s all I’ve got so far. I’ll keep updating this if I bump into anything new!

Configuring a custom SSL certificate in Zentyal

Sun, 25 Apr 2021 00:00:00 +1200

Zuper-quick post for when this bites me again in the future!

Tonight I upgraded from Zentyal 6.2 to Zentyal 7.0. Smooth as butter, everything went great.

Until I logged in, and my shiny Actual Paid Money SSL certificate had vanished, replaced by a self-signed commoner’s certificate. Piffle.

Instructions for installing a custom SSL certificate in Zentyal are actually kinda shaky, so here you go:

cd /var/lib/zentyal/conf/ssl/
mkdir old
mv * old
nano ssl.key
nano ssl.cert
cat ssl.cert ssl.key > ssl.pem
chmod 0400 ssl*

That’s it! Your Private Key lives in ssl.key, and your SSL certificate lives in ssl.cert.

Reboot, and job done.

Blocking bad extensions and extortion with iRedMail

Mon, 12 Apr 2021 00:00:00 +1200

Turns out this is my third Amavis article. I guess it’s just one of those systems.

Heads up, if you’re trying to do this, make sure you also read my other article about the “banned_files_lover” thing.

Today I’m on a mission to:

Drop all incoming Office ‘97 files (they’re predominantly malicious these days)
Drop all incoming Macro-enabled Office 2007+ files (there aren’t legitimate reasons to receive these in my scenario)
Drop any emails containing a .onion address
Drop any emails containing a bitcoin wallet

Dropping Attachments

The attachment block is easily handled by Amavis. In iRedMail on Debian, the configuration file is found at /etc/amavis/conf.d/50-user. Open your config file, and scroll down to the section where the $banned_filename_re variable is set. Insert the following line:

 qr'.\.(doc|dot|docm|docb|xls|xlm|xlt|xlsm|xlsb|
 xla|xlam|ppt|pps|pptm|potm|ppam|ppsm|sldm)$'i, # Office '97-2003 and Macro-enabled files
 qr'.\.(adn|accdb|accdr|accdt|accda|mdw|accde|mam|maq|mar|mat|
 maf|laccdb|ade|adp|mdb|cdb|mda|mdn|mdt|mdf|mde|ldb)$'i, # Microsoft Access files
 qr'^\.pub$', # Microsoft Publisher files

That will block all core Office ‘97-2003 files, as well as all Macro-enabled Office 2007-365 files.

By default, this will silently reject mails containing these attachments. If you want senders to receive a bounce message, search for the $final_banned_destiny variable and make sure it’s set to D_BOUNCE.

Dropping .onion and BTC

This part happens in Postfix, and it’s more-super-easy than the last bit. To configure Postfix’s body checks, edit /etc/postfix/body_checks.pcre. I simply added the following:

/(\w+\.onion)/ REJECT This mail server does not accept references to .onion addresses (${1})
/(\b(bc1|[13])[a-zA-HJ-NP-Z0-9]{25,39})/ DISCARD Bitcoin wallet detected (${1})

I used the REJECT directive for testing, but to be honest I don’t want to send bounces back to these people, so I changed to DISCARD after it was working. You can see the difference above.

The text after the REJECT directive is returned to the sender, and the text after the DISCARD directive is logged. The variable ${1} contains the detected string, and is appended to the response message.

That’s the end! Safety. Yes.

Acknowledgements

Thanks to Brad and Hamish for this post - Brad for the regex, Hamish for the idea, and both of them for the review!

Using Caddy to enable MTA-STS

Tue, 19 Jan 2021 00:00:00 +1300

About 7 months ago, I left Nginx and moved to Caddy. I’ve found it super easy, and have now experienced using it as a reverse proxy, a static site server (this one!), as well as a hosting a handy place for me to copy my favourite Kaomoji.

Update 2021-01-20: I had a bug! By default, Caddy serves this without a Content-Type header. However, the RFC specifies that the response should be explicitly Content-Type "text/plain". Thanks to klausenbusk for letting me know!

Note: the Caddyfile fragment to generate https://shrug.ericlight.com looks like this:

shrug.ericlight.com { respond "<meta charset='UTF-8'>¯\_(ツ)_/¯" }

Anyway, once I’d used Caddy’s respond directive, it was a simple step to take it forward and use it to serve my handy-dandy MTA-STS policy file!

Simply add this new Server block to your Caddyfile:

mta-sts.ericlight.com {

respond "version: STSv1
mode: testing
mx: in1-smtp.messagingengine.com
mx: in2-smtp.messagingengine.com
max_age: 86401"

header Content-Type "text/plain; charset=utf-8"

}

This simple Server block will get you a shiny HTTPS certificate, redirect any HTTP visitors to HTTPS, and will provide enquirers with your MTA policy file. (Obviously, replace my domain name and MX entries with your own.)

Sharp observers will notice that the MTA-STS policy file should be hosted at https://mta-sts.ericlight.com/.well-known/mta-sts.txt… but that’s the elegance of the Respond directive in Caddy: wherever you go at mta-sts.ericlight.com, you get exactly that response!

Example: https://mta-sts.ericlight.com/my/hovercraft/is/full/of/eels.txt

Okay, that was easy… the next part is just creating your three DNS records:

Record #1: An A record for mta-sts.ericlight.com, pointing to your Caddy server.
Record #2: A TXT record at _mta-sts.ericlight.com, reading v=STSv1; id=20210112. (Note the underscore! Also, the ID can be anything.)
Record #3: A TXT record at _smtp._tls.ericlight.com, reading v=TLSRPTv1; rua=mailto:tls-reports@targetdomain.com. (update that email address)

… There you go! Now you have a shiny new MTA-STS policy, and your Reporting email target (configured in DNS Record #3) will start reciving daily reports from sending mail servers, letting you know if your policy is working correctly.

Eventually (once you’re sure it’s all configured properly), you should change your testing policy to enforce. When you do this, make sure you update the ID in your TXT Record! This is how sending mail servers know that there has been a policy update.

Enjoy!

Getting TrueNAS (FreeNAS 12) to work with an APC NMC2 UPS (AP9631)

Thu, 14 Jan 2021 00:00:00 +1300

TrueNAS, and FreeNAS before it, has supported UPS via the NUT software package since forever. But most people using it seem to be using USB-connected UPS devices. I don’t have one of these. I’m dealing with an IPv4-based APC ups, specifically the Smart-UPS X 1500 (SMX1500RMI2UNC), with an AP9631 NMC2 card.

You’re probably here because you’re in the same boat: trying to set up a non-USB or Ethernet-based UPS under FreeNAS/TrueNAS, and you can’t. You’ve noticed the TrueNAS console will not shut up about errors which read something like:

nut plugin: nut_connect: upscli_connect (localhost, 3493) failed: Connection failure: Connection refused. (Or perhaps port 161)

“But why?”, you say. “I’ve given TrueNAS my UPS hostname, why is it always trying to talk to localhost? And what is this port 3493, when SNMP is 161?”

What’s happening is that upscli is trying to talk to the upsd damon on your TrueNAS server, which is expected to listen on port 3493 - it’s upsd which then does the talky with the SNMP driver, which in turn does the talky with the UPS. This is why the TrueNAS console logs look like it’s just shouting at itself.

Okay, let’s get into it:

UPS Configuration

First, you need to enable SNMP on your AP9631. This was pretty straightforward, but remember that you need to go:

Configuration -> Network -> SNMPv1 -> Access -> Enable
Configuration -> Netowrk -> SNMPv1 -> Access Control, and make sure the public community is set to be accessible from 0.0.0.0
Reboot your NMC2 card.

This configuration should work with the defaults that the NUT SNMP driver expects - I suggest you get it working like this for now, then tweak to SNMPv3 once you have a confirmed-working connection.

TrueNAS Configuration

Your UPS configuration menu is under TrueNAS -> Services -> UPS. Configure your TrueNAS UPS section like so:

General Options section:

Identifier: Up to you, but do yourself a favour and make it ups.
UPS Mode: Master
Driver: “Various ups 3 (various) SNMP - RFC 1628 (snmp-ups, experimental)” – once you save, this will simply display as snmp-ups$(various)
Port or Hostname: The hostname of the ups, hopefully something simple like ups.mydomain.local

Monitor section:

Leave the Monitor User, Monitor Password, Extra users, and Remote Monitor all blank/default for now - you can change them once the UPS is talking to you. Note I don’t believe it’s important to change the Monitor Password under this configuration, as upsd only listens on the loopback address.

Shutdown and Email sections

I shall leave in your capable hands.

Other Options:

Auxiliary Parameters (ups.conf): This is where you override the SNMP defaults, if desired. By default, NUT will use SNMPv1 and the ‘public’ community. See here for all config options availble for the SNMP driver. You could do something like this, if you want:

community=yaysecurity
snmp_version=3
privProtocol=AES

Auxiliary Parameters (upsd.conf): LISTEN 127.0.0.1 3493

That last line was the magic - this is the bit that makes upsd not only listen on port 161 (whyyyy), but also listen on 3493, which is where the rest of the UPS subsystem expects to find UPS data!

Other interesting tidbits:

Running upsc -l should list all UPS instances configured on your server. Since I’ve got one, called ‘ups’, I see:

# upsc -l
ups

If upsc -l gives you a Connection Refused error, check /usr/local/etc/nut/upsd.conf to make sure it’s listening on 3493. If not, add your auxiliary parameter in TrueNAS and restart the service.

# upsc ups
Error: Connection failure: Connection refused

# cat /usr/local/etc/nut/upsd.conf
LISTEN 127.0.0.1 161
LISTEN ::1 161

# sockstat -4 -l | grep upsd
uucp upsd 99934 6 tcp4 127.0.0.1:161 *:*

Above: upsd is definitely not listening on the expected port

NUT Config files: These live under /usr/local/etc/nut/. The folder is deleted and recreated every time you edit something in the TrueNAS GUI, so if you’re in that folder and change something in the GUI, you’ll need to do the whole cd /usr/local/etc/nut again. No you can’t do cd ../nut, because the whole inode is gone, so the OS doesn’t know where you are anymore.
Testing your UPS: Once upsd is listening on 3493, you should be able to run upsc ups, and it will query your UPS for data. Note that “ups” is your UPS identifier from the “General Options” section, also found as the first line of /usr/local/etc/nut/ups.conf:

# cat /usr/local/etc/nut/ups.conf
[ups]
 driver = snmp-ups
 port = ups.mydomain.local
 desc = 
 pollfreq = 15

# upsc ups
ambient.humidity: 0.00
ambient.temperature: 23.0
battery.charge: 100.00
battery.date: 05/15/2017
battery.packs: 0.00
battery.runtime: 1221.00
battery.runtime.low: 600
battery.voltage: 54.50
device.mfr: APC
device.model: Smart-UPS X 1500
[...]

I think that pretty much covers it. I hope this has been helpful to someone - at very least, I hope it’s helpful to future-me!

It’s been a little while since I’ve worked with UPS management software. I like to think it’s because my day job isn’t in infrastructure anymore, but also to be honest the whole UPS scene is fairly ugly. I recall speaking with an electrical engineer from Schneider a while ago, who jokingly said something like “creating a new UPS or CNC communications protocol is almost a rite of passage for any new engineer”. I can see why… when you’re new, and see something superbly awful, it’s quite compelling to try and fix it! But decades now of dependencies really do add up to be a burden.

Logging DNS queries, for both pfSense and Zentyal server

Sun, 06 Dec 2020 00:00:00 +1300

Logs of your client DNS queries can be a really good tool for incident response. I’ve finally got this implemented but it was much more of a struggle than I expected it to be, so here’s my story!

We’ve got a pfSense firewall running as a DNS fowarder, and a Zentyal server running BIND9 as the authoritative local server. The firewall rules block all UDP/53, so all DNS queries go either directly to the firewall, or (more commonly) to the Zentyal server for resolution. This means we have two places that resolve DNS queries which we want to forward off to our syslog server.

pfSense

Unbound is a super simple DNS forwarder, and the configuration is wonderfully straightforward. It did take me quite a while to figure it out, but in the end it was a simple oversight on my part.

To get pfSense/Unbound to forward DNS queries to your syslog server, simply open the Services -> DNS Resolver page, click ‘Display Custom options’, and add these two lines:

:::ini
server:
log-queries: yes

I spent hours fiddling before I realised I was missing the empty “server:” directive. -__-

This will log all DNS queries that Unbound deals with. These logs go to the standard internal pfSense log; if you want them to be replicated off to an external syslog server, go Status -> System Logs -> Settings. Scroll down to the “Remote Log Servers” section and add your syslog server there. (I’m not covering syslog listener config in this particualar post!)

Zentyal

Yeah this one was harder to figure out. Because Zentyal overwrites the config at every damn opportunity, you can’t just edit /etc/bind/named.conf.options, like what you find in most articles… it may work for a moment, but it’ll be overwritten.

The real answer to editing Zentyal’s config files is outlined at https://doc.zentyal.org/en/appendix-c.html#stubs:

:::bash
cp /usr/share/zentyal/stubs/dns/named.conf.options.mas /etc/zentyal/stubs/dns/named.conf.options.mas

Once you’ve copied the stub into a place where it won’t be clobbered, edit it and zoom right down to the bottom line, which looks like:

:::ini
logging { category lame-servers { null; }; };

This is the line you need to replace. Now, you can go as ham as you want here. I’ve actually used the sample provided by ISC themselves, which you can find at https://kb.isc.org/docs/aa-01526. However, that’s really robust and possibly more than you may need. A bare minimum would probably look like:

:::ini
logging { 
 channel queries_log {
 syslog named;
 print-time yes;
 print-category yes;
 print-severity yes;
 severity info;
 };
};

Note - when I tried to use the ISC sample wholesale, I needed to change a couple small things:

Changed all /var/named/log output destinations to /var/log/named
The lines for category zoneload and category trust-anchor-telemetry need to be removed for Zentyal 6.2’s version of BIND9
I needed to create /var/log/named, and make sure it was writeable by the bind user

I think that covers it, I hope this helps someone! I’ve actually even taken the time today to create a Pull Request in the Zentyal GitHub, we’ll see if it ends up getting merged!

iRedMail, SpamAssassin, and Lynis

Mon, 02 Nov 2020 00:00:00 +1300

I really like iRedMail, and I also really like Lynis.

However, they don’t exactly like each other… or, more accurately, some of Lynis’ recommendations can cause a couple iRedMail components to fail. Today we’re talking about SpamAsassin.

One of the suggestions from Lynis is to turn off the ’execute’ bit on compilers for users who aren’t either the owner or in the owner group (the ‘other’ execute bit). For example:

:::bash
root@server:/# chmod o-x /usr/bin/as
root@server:/# chmod o-x /usr/bin/gcc

Easy peasy! But once you do this, you might start getting the following in your daily iRedMail Cron reports:

:::text
/etc/cron.daily/spamassassin:
/bin/sh: 1: x86_64-linux-gnu-gcc: Permission denied
make: *** [Makefile:346: body_0.o] Error 126
command 'make PREFIX=/tmp/.spamassassin23046Zmmrr9tmp/ignored INSTALLSITEARCH=/var/lib/spamassassin/compiled/5.028/3.004002 >>/tmp/.spamassassin23046Zmmrr9tmp/log' failed: exit 2

There are probably a bunch of ways to fix this. My way, I’m sure, is not the best way… however it was quick and easy, and it worked.

:::bash
root@server:/# chgrp debian-spamd /usr/bin/as
root@server:/# chgrp debian-spamd /usr/bin/gcc
root@server:/# runuser -l debian-spamd -c sa-compile

Look ma, no more errors!

This solution really only works for me because debian-spamd is the only non-root user that calls these compilers. If I had another user which needed to call them, I’d have to come up with a better fix. But for a standalone iRedMail server, this does the trick!

iRedMail: Daily user unknown entries from backup_sogo.sh

Sun, 01 Nov 2020 00:00:00 +1300

If you’ve been running iRedMail for a while, eventually you’ll probably start seeing ‘user unknown’ events in your daily logs:

:::text
* Backup all users' data under /var/vmail/backup/sogo/2020/11/01
<0x0x5574a64c36b0[SOGoCache]> Cache cleanup interval set every 300.000000 seconds
<0x0x5574a64c36b0[SOGoCache]> Using host(s) '127.0.0.1' as server(s)
2020-11-01 09:29:12.784 sogo-tool[29749:29749] user 'abdulm' unknown
2020-11-01 09:29:12.786 sogo-tool[29749:29749] user 'bent' unknown
2020-11-01 09:29:12.786 sogo-tool[29749:29749] user 'brettr' unknown
2020-11-01 09:29:12.786 sogo-tool[29749:29749] user 'catalinar' unknown
2020-11-01 09:29:12.787 sogo-tool[29749:29749] user 'clinth' unknown
2020-11-01 09:29:12.787 sogo-tool[29749:29749] user 'danield' unknown
2020-11-01 09:29:12.787 sogo-tool[29749:29749] user 'dannyn' unknown
2020-11-01 09:29:12.788 sogo-tool[29749:29749] user 'darcyk' unknown
2020-11-01 09:29:12.788 sogo-tool[29749:29749] user 'davidl' unknown
* Compress backup files.

This is because removal of an iRedMail user doesn’t remove the corresponding SOGo user data. You can take care of this with sogo-tool:

:::bash
root@server:/# sogo-tool remove abdulm@<domain.xyz> bent@<domain.xyz> brettr@<domain.xyz>

… etc. Once you’re done, run the backup again to make sure you’ve got them all:

:::bash
root@server:/# /bin/bash /var/vmail/backup/backup_sogo.sh
* Backup all users data under /var/vmail/backup/sogo/2020/11/01
<0x0x563fd0e2b6b0[SOGoCache]> Cache cleanup interval set every 300.000000 seconds
<0x0x563fd0e2b6b0[SOGoCache]> Using host(s) '127.0.0.1' as server(s)
* Compress backup files.

ZFS zpool vanishing after upgrading ProxmoxVE 5.4 to 6.2

Sat, 23 May 2020 00:00:00 +1200

Performing a major version upgrade is never pleasant. I’ve been using ProxmoxVE for about ten years now though, and it’s consistently done a fantastic job. Since it’s based on Debian, all the upgrades are done with a simple apt update && apt upgrade, with a variety of steps in the middle to point to new repositories, etc. Nothing out of the ordinary, nothing scary.

So I’ve got two servers - one is an old one, with only a couple testing VM’s residing on it. I go ahead and work through the 5.x to 6.x upgrade guide, and everything Just Works. No problem with that one.

When I moved onto the other server I had a couple different things I noticed. For example, when trying to upgrade Corosync 2.x to 3.x (a prerequisite for the OS upgrade), I noticed it was trying to uninstall things like corosync-pve, which really didn’t seem right.

After comparing dpkg -l contents between the two servers, I learned that corosync-pve is actually just a transitional package - completely replaced by the new corosync package in version 3. And although the server was trying to uninstall corosync-pve, it was then immediately installing corosync… so no problem.

Things continued to be mildly alarming during the upgrade itself, but not actually toooo bad, as all the removed packages were being replaced. This one, for example:

dpkg: pve-libspice-server1: dependency problems, but removing anyway as you requested:
 pve-qemu-kvm depends on pve-libspice-server1 (>= 0.12.5-1); however:
 Package pve-libspice-server1 is to be removed.
 spiceterm depends on libspice-server1 (>= 0.12.2); however:
 Package libspice-server1 is not installed.
 Package pve-libspice-server1 which provides libspice-server1 is to be removed.

(Reading database ... 65255 files and directories currently installed.)
Removing pve-libspice-server1 (0.14.1-2) ...

… was replaced with spiceterm. No worries.

However things went sharply downhill following reboot. All of my VM images were missing! I had a ZFS zpool that should have been at /VM_Local_zpool - that folder existed, and contained an ./images folder, but that was completely empty.

I quickly checked the storage information in Proxmox, and the result was… not good:

… Really, not good. :-|

So I SSH’ed into the server and had a quick look at my list of ZFS stores, and discovered that my dataset was entirely absent:

# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 83.8G 23.8G 96K /rpool
rpool/ROOT 3.65G 23.8G 96K /rpool/ROOT
rpool/ROOT/pve-1 3.65G 23.8G 3.65G /
rpool/data 71.6G 23.8G 71.6G /rpool/data
rpool/swap 8.50G 25.4G 6.85G -

# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 111G 82.1G 28.9G - - 66% 73% 1.00x ONLINE -

A bit of Googling led me to the zfs import command, which resulted in the first piece of good news of the night:

# zpool import

 pool: VM_Local_zpool
 id: 6614373363984244305
 state: ONLINE
 status: Some supported features are not enabled on the pool.
 action: The pool can be imported using its name or numeric identifier, though
	some features will not be available without an explicit 'zpool upgrade'.
 config:

	VM_Local_zpool ONLINE
	 mirror-0 ONLINE
	 wwn-0x50014ee0aeee96ef ONLINE
	 wwn-0x50014ee05998ee25 ONLINE

It exists! And it’s … online? But where? I don’t get it. I went ahead and tried importing it, but got a “directory not empty” error:

# zpool import VM_Local_zpool 
cannot mount '/VM_Local_zpool': directory is not empty

Now, if I’d been paying attention there, I would have realised that zpool list now contained my VM_Local_zpool:

# zfs list
NAME USED AVAIL REFER MOUNTPOINT
VM_Local_zpool 1.16T 611G 1.16T /VM_Local_zpool
rpool 83.8G 23.8G 96K /rpool
rpool/ROOT 3.65G 23.8G 96K /rpool/ROOT
rpool/ROOT/pve-1 3.65G 23.8G 3.65G /
rpool/data 71.6G 23.8G 71.6G /rpool/data
rpool/swap 8.50G 25.4G 6.85G -

From this point, all I had to do was empty /VM_Local_zpool, and try to remount the zpool. But there’s a second trick here.

ProxmoxVE has a pretty clear idea of How Storage Should Look, so every time it scans a storage device, is makes sure there’s an ./images folder in any storage enabled for VM Images. It also makes sure there’s a ./dump folder on any storage enabled as a backup target, and a ./template folder on any storage enabled for ISO images. (Reference: https://pve.proxmox.com/wiki/Storage:_Directory)

So, when I removed that /VM_Local_zpool/images folder, Proxmox just… went ahead and recreated it almost immediately, so I still couldn’t mount the zpool there.

In the end, I managed to get my zpool remounted by pairing the rmdir and mount commands together on the same line:

# rmdir /VM_Local_zpool/images/
# zfs mount VM_Local_zpool
cannot mount '/VM_Local_zpool': directory is not empty
# rmdir /VM_Local_zpool/images/ && zfs mount VM_Local_zpool
# cd VM_Local_zpool/
/VM_Local_zpool# ls
backup	images

… and there we go! My mount point, and all my VM images, are back unscathed!

And just for future reference - I saw a post on the FreeNAS forums that I should the status of the zfs-import-cache service, and enable it if it wasn’t started by default… however it was fine:

# systemctl status zfs-import-cache.service
● zfs-import-cache.service - Import ZFS pools by cache file
 Loaded: loaded (/lib/systemd/system/zfs-import-cache.service; enabled; vendor preset: enabled)
 Active: active (exited) since Sat 2020-05-23 20:46:54 NZST; 3min 26s ago
 Docs: man:zpool(8)
 Process: 1819 ExecStart=/sbin/zpool import -c /etc/zfs/zpool.cache -aN (code=exited, status=0/SUCCESS)
 Main PID: 1819 (code=exited, status=0/SUCCESS)

May 23 20:46:54 ~ systemd[1]: Starting Import ZFS pools by cache file...
May 23 20:46:54 ~ zpool[1819]: no pools available to import
May 23 20:46:54 ~ systemd[1]: Started Import ZFS pools by cache file.

Using Fail2Ban to insta-block malicious hits

Tue, 07 Apr 2020 00:00:00 +1200

Okay so a few days ago I posted about adopting Caddy.

Of course, anyone who hosts anything knows that accessible services on the internet will start getting hit by scanners within literal moments of being accessible. On the very first night of having my site live on the new VPS, Logwatch informed me of a bunch of attempts to access wp-admin.php. Sigh.

Well, this entire site is built with my favourite static site generator Pelican. That means I don’t have any PHP on my site whatsoever. Why not just… block anyone that tries to open a PHP page at ericlight.com? Fail2Ban to the rescue.

Two things we need: A Fail2Ban filter, and a Fail2Ban jail.

The Filter

nano /etc/fail2ban/filter.d/caddy-php.conf (note, the name of this file must be the same as the identifier in your jail.local file)

[Definition]
failregex = ^<HOST>.*\.php(\s|\?.*)HTTP.*$
ignoreregex =

… that will catch anything in your logs such as:

nn.nn.nn.nn - - [07/Apr/2020:23:20:39 +1200] "GET /wp-admin.php HTTP/2.0" 404 0
nn.nn.nn.nn - - [06/Apr/2020:10:12:24 +1200] "GET /index.php?s=index/%5Cthink%5Capp/invokefunction&function=call_user_func_array&vars%5B0%5D=phpinfo&vars%5B1%5D%5B%5D=1 HTTP/1.1" 404 0
nn.nn.nn.nn - - [06/Apr/2020:12:51:31 +1200] "GET /wp-login.php HTTP/1.1" 404 0

The Jail

nano /etc/fail2ban/jail.local

[caddy-php]
port = http,https
logpath = /var/log/caddy/access.log
enabled = true
maxretry = 1

Note I used maxretry = 1 there. This means if an IP triggers that filter a single time, they’ll be blocked immediately… So you probably don’t want to do this just willy-nilly.

The Result

With those two parts done (and the obligatory service fail2ban reload, of course), you should find that your web server immediately bans any host that tries to load a .php file from your website!

# tail -f /var/log/caddy/access.log /var/log/fail2ban.log
==> /var/log/caddy/access.log <==
1xx.1xx.1xx.11 - - [07/Apr/2020:23:25:55 +1200] "GET /arse.php HTTP/1.1" 404 0
1xx.1xx.1xx.6 - - [07/Apr/2020:23:25:57 +1200] "GET /arse.php?1234 HTTP/1.1" 404 0
1xx.2xx.6x.1xx - - [07/Apr/2020:23:26:03 +1200] "GET /blargh.php HTTP/2.0" 404 0

==> /var/log/fail2ban.log <==
2020-04-07 23:25:55,482 fail2ban.filter [452]: INFO [caddy-php] Found 1xx.1xx.1xx.11 - 2020-04-07 23:25:55
2020-04-07 23:25:55,551 fail2ban.actions [452]: NOTICE [caddy-php] Ban 1xx.1xx.1xx.11
2020-04-07 23:25:58,309 fail2ban.filter [452]: INFO [caddy-php] Found 1xx.1xx.1xx.6 - 2020-04-07 23:25:57
2020-04-07 23:25:58,782 fail2ban.actions [452]: NOTICE [caddy-php] Ban 1xx.1xx.1xx.6
2020-04-07 23:26:03,627 fail2ban.filter [452]: INFO [caddy-php] Found 1xx.2xx.6x.1xx - 2020-04-07 23:26:03
2020-04-07 23:26:04,010 fail2ban.actions [452]: NOTICE [caddy-php] Ban 1xx.2xx.6x.1xx

Great success! You can use the banTime directive to adjust how long these blocks should last for - I’ve got mine set up to block for an hour.

Edit 2020-04-10:

Eh I realised that my RegEx missed these lovely things:

[ip redacted] - - [08/Apr/2020:16:14:10 +1200] "GET /wp-config.php.new HTTP/1.1" 404 0
[ip redacted] - - [08/Apr/2020:16:14:14 +1200] "GET /wp-config.php.old HTTP/1.1" 404 0
[ip redacted] - - [08/Apr/2020:16:14:17 +1200] "GET /wp-config.php.bak HTTP/1.1" 404 0
[ip redacted] - - [08/Apr/2020:16:14:19 +1200] "GET /wp-config.php.backup HTTP/1.1" 404 0
[ip redacted] - - [08/Apr/2020:16:14:22 +1200] "GET /wp-config.php.save HTTP/1.1" 404 0

… so yeah I just changed it to drop anything with .php in the URL. I’ll try to remember not to post any articles with .php in the URL lol.

Credits

Thanks to Phage and Xyphoid for the help in fine-tuning my rusty RegEx!

Moving to the Caddy web server

Sun, 05 Apr 2020 00:00:00 +1300

For the last couple of years I’ve been running this site, as well as my friend’s site (Under The Umbrella) on Nginx. Recently my VPS host decided to do away with their cheapest tier, so instead of doubling my annual cost, I hopped onto https://www.lowendbox.com and found myself a replacement Cheaps McGee VPS to host this.

Well, a major change like that is a great time to learn about something new, so I took the opportunity to get started with Caddy. If you don’t already know about Caddy, it’s a fast, simple, clean web server. It’s written in Go, so it’s both fast and memory safe. And hey, it’s super simple.

I’m not going to go into a whole lot of detail about setting up Caddy - there are enough tutorials out there already, and really I got all the info I needed from the website. But here are some particularly notable bits:

The Caddyfile

This lives in /etc/caddy/Caddyfile:

ericlight.com, www.ericlight.com {

 file_server
 root * /var/www/ericlight.com

 import /etc/caddy/caddy_security.conf

 log {
 output file /var/log/caddy/access.log
 format single_field common_log
 }
}

undertheumbrella.co.nz, www.undertheumbrella.co.nz {

 file_server
 root * /var/www/undertheumbrella.co.nz

 import /etc/caddy/caddy_security.conf

 log {
 output file /var/log/caddy/utu_access.log
 format single_field common_log
 }
}

And, /etc/caddy/caddy_security.conf contains:

header {
 Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
 X-Xss-Protection "1; mode=block"
 X-Content-Type-Options "nosniff"
 X-Frame-Options "DENY"
 Content-Security-Policy "upgrade-insecure-requests"
 Referrer-Policy "strict-origin-when-cross-origin"
 Cache-Control "public, max-age=15, must-revalidate"
 Feature-Policy "accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'self'; camera 'none'; encrypted-media 'none'; fullscreen 'self'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture *; speaker 'none'; sync-xhr 'none'; usb 'none'; vr 'none'"
}

** Update 2022-01-05:** Previously I’d used header / { above; that should have been simply header {. Thanks @JoHoffmann8 for pointing this out! It’s also worth mentioning that Caddy are deprecating the common_log log format, which makes me sad in one way, but I do get it - the Caddy json log file format is far richer, but I liked the easily-ingested syslog format. ¯\_(ツ)_/¯

Ok so here’s the thing. Caddy really seems to implement Python’s ethos of “Batteries Included”. The above contents are enough on their own to:

Host two separate static websites
Offer two subdomains for each of these websites
Manage the entire certificate creation and renewal process from Let’s Encrypt for two sites, plus subdomains.
Get an A+ rating on both SSL Labs and SecurityHeaders.io(!!)
And, of course, zoink all the logs into separate files under /var/log/caddy

Creating a Caddy Service file

UPDATE 2020-05-07: With the release of Caddy 2.0, it appears a regular dpkg -i caddy.deb will take care of creating the caddy.service file

If you’re running Debian, you’ll need to create yourself a service file for systemd, so you can get your server to launch Caddy on boot. I got mine from https://github.com/caddyserver/dist/tree/master/init:

/etc/systemd/system/caddy.service:

# This service file requires the following:
#
# 1) Group named caddy:
# $ groupadd --system caddy
#
# 2) User named caddy, with a writeable home folder:
# $ useradd --system \
# --gid caddy \
# --create-home \
# --home-dir /var/lib/caddy \
# --shell /usr/sbin/nologin \
# --comment "Caddy web server" \
# caddy
#
# 3) Caddyfile at /etc/caddy/Caddyfile that is
# readable by the caddy user
#

[Unit]
Description=Caddy Web Server
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --config /etc/caddy/Caddyfile --environ
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

Preventing scans

Everything above is already pretty secure - Caddy is really good at making security super easy. On top of that, Caddy is the only service hosted on this box, there’s no dynamic code (all raw HTML and CSS, thanks to Pelican), and the only things listening to the internet are SSH and Caddy itself. But even then, I get tired of seeing hundreds of scan reports every day. Fail2Ban to the rescue.

/etc/fail2ban/filter.d/caddy-4xx.conf:

[Definition]
failregex = ^<HOST>.*"(GET|POST).*" (404|444|403|400) .*$
ignoreregex =

/etc/fail2ban/jail.local:

[caddy-4xx]
port = http,https
logpath = /var/log/caddy/access.log
 /var/log/caddy/utu_access.log
enabled = true
banTime = 3600
findTime = 600
maxretry = 5

Fin!

And that’s all! I had another tweak or two to my Pelican Makefile, to point rsync to the right server, but overall that was an incredibly simple process. The Caddy team have done a spectacular job.

Fail2Ban+Nginx (blocking repeated 404's, etc)

Mon, 16 Mar 2020 00:00:00 +1300

After hosting a mail server for a few years, I’ve gotten tired of seeing alllll the 404 attempts in my daily logwatch. Fail2Ban can help here really well, and it turns out to be really easy.

Start with nano /etc/fail2ban/filter.d/nginx-4xx.conf (I’m counting on your running Debian and having things in default locations here), and enter the following:

[Definition]
failregex = ^<HOST>.*"(GET|POST).*" (404|444|403|400) .*$
ignoreregex =

Note: Make sure you use a capital ‘D’ in Definition there.

Now, edit your /etc/fail2ban/jail.conf, and add these lines:

[nginx-4xx]
enabled = true
port = http,https
logpath = /var/log/nginx/access.log
maxretry = 3

Of course, modify your maxretry as desired. Refresh your Fail2Ban rules with service fail2ban restart, and it should now be on the lookout for repeated 4xx errors.

You can confirm it’s working with: fail2ban-client status nginx-4xx:

Status for the jail: nginx-4xx
|- Filter
| |- Currently failed:	2
| |- Total failed:	9
| `- File list:	/var/log/nginx/access.log
`- Actions
 |- Currently banned:	2
 |- Total banned:	2
 `- Banned IP list: 123.232.123.101 123.232.123.102

Success!

Worth mentioning… for the first while I thought it wasn’t working, but I realised I was trying to trigger the rule from on the same LAN, and …

[DEFAULT]
# time is in seconds. 3600 = 1 hour, 86400 = 24 hours (1 day)
findtime = 3600
bantime = 3600
maxretry = 5
ignoreip = 127.0.0.1 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16

… I was hitting it from within the ignoreip range. 😬

And of course, I can’t finish off a blog without giving credit where it’s due. I got this all from GitHub user AysadKozanoglu, here: https://gist.github.com/AysadKozanoglu/1335735272fb3b00a03bd3eea22af818

sa-compile failing during upgrade

Sun, 15 Mar 2020 00:00:00 +1300

I just spent an hour troubleshooting the most ridiculous thing. I guess to help people search, I’d describe this as “dpkg failing at sa-compile in Debian 10” but that’s really not a good picture of what’s happening here…

Setting up sa-compile (3.4.2-1~deb9u3) ...
Running sa-compile (may take a long time)
/bin/sh: 1: x86_64-linux-gnu-gcc: Permission denied
make: *** [body_0.o] Error 126
command 'make PREFIX=/tmp/.spamassassin22062Ifq5yDtmp/ignored INSTALLSITEARCH=/var/lib/spamassassin/compiled/5.024/3.004002 >>/tmp/.spamassassin22062Ifq5yDtmp/log' failed: exit 2
dpkg: error processing package sa-compile (--configure):
subprocess installed post-installation script returned error exit status 25

<-- snip -->

Errors were encountered while processing:
 sa-compile
E: Sub-process /usr/bin/dpkg returned an error code (1)

In my defence, it seemed similar to a recurring issue I’ve had on this server in the past, (which I now promise to address in a separate post). That’s why I ended up on a red herring hunt.

Anyway, I spent a bunch of time trying to troubleshoot dpkg and Python, before realising that a few months ago I’d installed Lynis and worked through a bunch of the hardening recommendations.

… One of which is restricting the execute permissions on /usr/bin/gcc. Which is a symlink to /usr/bin/gcc-6. Which is a symlink to x86_64-linux-gnu-gcc-6.

root@x:/usr/bin# ls -l x86_64-linux-gnu-gcc-6
-rw-r--r-- 1 root root 949016 Feb 15 2018 x86_64-linux-gnu-gcc-6

Yes. I was experiencing problems compiling things because I’d removed execute permissions for all users on the compiler I needed, and I’d spent an hour ignoring the “Permission denied” error that told me where to look.

Fixed with a simple chmod 744 /usr/bin/x86_64-linux-gnu-gcc-6.

Zentyal 6.0 to 6.1 upgrade getting stuck

Tue, 17 Dec 2019 00:00:00 +1300

So I faced a little challenge with a Zentyal server the other day. I was upgrading from ye olde 6.0 to 6.1, when everything just stopped. I let it sit in the corner for about an hour or so, but it never picked up the thread. All the services were still live, so I logged in to have a look.

(Note, my DNS server is named RIMU; yours may be something else!)

Running ps aux, I discovered this line:

:::bash
sh -c /usr/bin/sudo -p sudo: /var/lib/zentyal/tmp/x2M7gkZVvm.cmd 2> /var/lib/zentyal/tmp/stderr

So, I had a quick look at the contents of that stderr file:

:::bash
# cat /var/lib/zentyal/tmp/stderr 
Password has expired
dns-RIMU@ad.ericlight.com's Password:

And what does that temp .cmd file contain?

:::bash
# cat /var/lib/zentyal/tmp/x2M7gkZVvm.cmd 
kinit -k -t /var/lib/samba/private/dns.keytab dns-RIMU

Running that kinit command indeed prompts for a password reset, but the interesting thing is that samba-tool shows me this password shouldn’t expire:

:::bash
# pdbedit -u dns-RIMU -v | grep change
Password can change: Wed, 31 Oct 2018 21:47:30 NZDT
Password must change: never

… riiiiiiight, that’s a bit interesting. And yet I’m still being prompted to set a new password. I used samba-tool to remind samba that this password shouldn’t expire:

:::bash
# samba-tool user setexpiry dns-RIMU --noexpiry
Expiry for user 'dns-RIMU' disabled.

And now…

:::bash
# pdbedit -u dns-RIMU -v | grep -i change
Password can change: Wed, 31 Oct 2018 21:47:30 NZDT
Password must change: Tue, 19 Jan 2038 16:14:07 NZDT

Now I can run kinit against dns-RIMU perfectly fine, and indeed the Zentyal upgrade succeeded!

New things I didn't know about WireGuard

Thu, 27 Dec 2018 00:00:00 +1300

This is part of my brief series on WireGuard. I’m pretty enamoured with WireGuard and the way it works, and I’ve been using it pretty seamlessly for over a year now. I’ve learned a couple things that weren’t immediately obvious though, so I’m documenting them here.

Easy Provisioning

Samuel Holland mentioned an interesting trinket, in his post at https://lists.zx2c4.com/pipermail/WireGuard/2018-December/003703.html:

"[…] WireGuard will ignore a peer whose public key matches the interface’s private key. So you can distribute a single list of peers everywhere."

You can combine this with wg addconf like this:

Each peer has its own /etc/WireGuard/wg0.conf file, which only contains it’s [Interface] section
Each peer also has a shared /etc/WireGuard/peers.conf file, which contains all the peers
The wg0.conf file also has a PostUp hook, calling wg addconf /etc/WireGuard/peers.conf

It’s up to you to decide how you want to share the peers.conf, be it via a proper orchestration platform, something much more pedestrian like Dropbox, or something kinda wild like Ceph. I dunno, but it’s pretty great that you can just wildly fling a peer section around, without worrying whether it’s the same as the interface.

Setting Private Key from a file

Another piece of learning, courtesy of Samuel Holland, at https://lists.zx2c4.com/pipermail/WireGuard/2018-December/003702.html.

You can read in a file as the Private Key by doing something like:

PostUp = wg set %i private-key /etc/WireGuard/wg0.key

When AppArmor eats your breakfast

Mon, 19 Mar 2018 00:00:00 +1300

I’ve knocked heads with AppArmor a few times now. To be quite honest with myself, I think it’s primarily because I install it, enable all the plugins, and then forget it’s there until there’s a problem.

TL;DR:

On a fully-updated Zentyal 5.0 system running DHCP, the AppArmor profile in /etc/apparmor.d/usr.sbin.dhcpd will prevent isc-dhcp-server from restarting itself after an upgrade.

More search-type words: Zentyal dhcp server doesn’t start again after upgrade. isc-dhcp-server graceful shutdown, but no restart.

This frustration-laden, Google-friendly equivalent of speaking slowly and clearly should illustrate how impossible it was for me to find guidance on this.

To fix:

# aa-complain /etc/apparmor.d/usr.sbin.dhcpd

Okay, now that the TL;DR is finished, here’s the rest of the story:

Unfortunately there’s always a problem with AppArmor eventually.

Even worse, the problem is often silent. I’ll just find that, for example, NONE of my man pages work. Or logrotate will fail. For example, here’s what happens when you enable aa-enforce mode on your friendly resident user manual:

 $ man
 What manual page do you want?
 $ sudo aa-enforce /etc/apparmor.d/usr.bin.man
 Setting /etc/apparmor.d/usr.bin.man to enforce mode.
 $ man
 $man: can't open the manpath configuration file /etc/manpath.config

If I hadn’t enabled aa-enforce immediately before that, what would you have done? You would have gone ahead, made sure that manpath.config existed, made sure it wasn’t corrupted, maybe tried running man as root… maybe even copied a manpath.config file from a known-working computer. But, hidden within dmesg is our old friend:

[149929.763064] type=1400 audit(1521449052.257:111): apparmor="DENIED" operation="open" profile="/usr/bin/man" name="/etc/manpath.config" pid=22633 comm="man" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

And that example is actually better than it used to be - quite a few times in the past four years, I’ve received this:

$ man
$

… from a fully-updated Debian system.

This makes me sad.

Or, in one particular case, ISC-DHCP-SERVER of all things will launch perfectly well, but will fail to restart after a package update, and the first thing I’ll notice is “o hai all the things is broken halp plz”. Something like that.

When you’re troubleshooting DHCP, you’ll probably check systemctl, leases table, maybe /var/log … But when you get around to checking dmesg, you may notice the root of the problem:

From dmesg -T:

[Sat Mar 17 03:44:28 2016] audit_printk_skb: 18 callbacks suppressed

Considering the update occurred at 03:44am, this is probably our problem. I can’t see for sure, but I’m pretty certain those 18 suppressed callbacks are filled with “DENIED” log lines.

The fix is simple:

# aa-complain /etc/apparmor.d/usr.sbin.dhcpd 
Setting /etc/apparmor.d/usr.sbin.dhcpd to complain mode.

And your isc-dhcp-server should now restart gracefully after it’s scheduled upgrades!

Installing PowerChute Network Shutdown on ProxmoxVE

Tue, 27 Feb 2018 00:00:00 +1300

This is going to be one of those posts where I just dump stuff. Power failures are shite, and if your server isn’t talking with your UPS, it won’t shut down gracefully. This is the story of my trying to get APC’s PowerChute Network Shutdown working on a ProxmoxVE 5.1 environment.

Note I opted out of using either nut or apcupsd, because I had a nice fancy Network Management Card (NMC2 / Schneider AP9631) available. Also, I wanted to shut down more than a single server, so a USB or Serial cable wouldn’t do the trick.

Install Java Runtime Environment: apt install openjdk-8-jre
Download PCNS onto your Proxmox server - http://www.apc.com/shop/us/en/categories/power/uninterruptible-power-supply-ups-/ups-management/powerchute-network-shutdown/N-auzzn7
Extract the download: tar -xvf pcns420Linux-x86-64.tar.gz
cd ./Linux_x64
sudo su
./install.sh
When prompted for your Java location, use /usr/lib/jvm/java-8-openjdk-amd64/jre/bin
I didn’t enable SMTP

Results in:

 root@ ~/Linux_x64# ./install.sh 
 ------------------------------------------------------------------
 PowerChute Network Shutdown 4.2.0 for Linux
 Copyright (c) 1999-2016 Schneider Electric.
 All Rights Reserved.
 ------------------------------------------------------------------

 OS=Linux

 Initializing ...

 Press any key to display End User License Agreement
 <--snip-->
 Do you agree to the above license terms? [yes or no]
 yes

 Please enter the installation directory or press enter to install to the default directory (/opt/APC/PowerChute):


 Are you sure you want to install PCNS to /opt/APC/PowerChute [Yes|No]? 
 yes
 PCNS will be installed to /opt/APC/PowerChute

 Please enter java directory if you want to use your system java (example:/usr/local/bin/jre/jre1.8.0_91) or press enter to install the bundled Java:
 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin 

 Checking version of Java ...
 Detected Java Version: 1.8.0.151
 Acceptable version

 openjdk version "1.8.0_151"
 OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-1~deb9u1-b12)
 OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

 JAVA_DIR=/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/

 Copying the installation files ...
 Extracting PCNS files ...
 PCNS is extracted to /opt/APC/PowerChute
 Configuring startup files ...
 Startup script=/etc/rc.d/init.d/PowerChute
 Updating Linux symbolic link ...
 Configure Firewall
 Configuring uninstall script ...
 Setup the m11.cfg file

 Enable SNMP Support [Yes|No]? 
 no
 SNMP Not Enabled

 PowerChute Network Shutdown, v4.2.0
 Copyright (c) 1999-2016, Schneider Electric. All Rights Reserved.
 Startup completed.


 Installation has completed.
 PowerChute Network Shutdown can be accessed through your browser at https://<your_server_ip_address>:6547
 Please complete the configuration wizard so that PowerChute Network Shutdown can protect your server.

Once you’re finished with that, navigate to the new web portal on your server (second-to-last line in the output above), and complete the setup of your PCNS instance!

When you’re configuring PCNS itself, you’ll be prompted for the User Name, Password, and Authentication Phrase for the UPS. These are the user details you use to log into your NMC.

By default, the Authentication Phrase is admin user phrase, but you should totally change this. Assuming you’ve updated the firmware on your NMC, you can set the Authentication Phrase at Configuration -> Shutdown.

Setting up your outlet groups and shutdown times are left as an exercise to the reader. :)

Fix Dell Keyboard Backlight under Debian

Thu, 11 Jan 2018 00:00:00 +1300

My personal laptop is an old Dell Latitude E6410. One of the things I actually love about it, is the fact that the keyboard has a backlight.

However, because Linux, sometimes that backlight just stops working. When this happens, you’ll see error messages in dmesg, such as:

dell_wmi: Unknown key with type 0x0011 and code 0x01e2 pressed

If this happens to you, just run this command:

$ sudo echo 7 > /sys/devices/platform/dell-laptop/leds/dell\:\:kbd_backlight/brightness

You can echo a higher number if you want your keyboard brighter, but I found 7 to be more than bright enough.

When the problem is DNS: FreeNAS and NFS

Sun, 07 Jan 2018 00:00:00 +1300

I discovered a while ago that NFS requires DNS to function correctly: https://redmine.ixsystems.com/issues/4983

That’s somewhat annoying, because when your DNS server goes down (and your backups are stored on your FreeNAS server and accessed over NFS), is precisely the time when you really want your backups to be accessible.

However, turns out it doesn’t aaaaactuallyyyyy need DNS… it needs name resolution. Specifically, FreeNAS just needs to be able to resolve it’s own hostname. Cue the handy-dandy hosts file.

Under Network -> Global Configuration -> Host name data base, add your details:

192.168.88.5 backupserver backupserver.local backupserver.ad.mydomain.com

If you have a secondary IP address, duplicate the above line and replace the IP as appropriate.

Obviously, replace the hostname and your internal domain name with whatever you use. However, be aware that the .local FQDN is actually required, even if you use a different domain name.

Making MindTouch's Dekiwiki work on Debian Stretch

Sun, 17 Sep 2017 00:00:00 +1200

We had accumulated a certain amount of technical debt, due to a Wiki solution that was selected a few years ago: Dekiwiki, by MindTouch. Unfortunately a few months after implementation, MindTouch Core (which Dekiwiki builds on) was well-and-truly deprecated back in 2013.

It all happened before my time, but it seems as if Dekiwiki came as a pre-built VMware Appliance, based on Debian Etch (Debian 4.0; released in 2007). After giving our Dekiwiki environment some serious side-eye for a while, I finally decided to get my hands dirty and try to upgrade it.

Jedd, over at Jeddi.org, wrote a really helpful post back in 2015 about his experiences updating the ageing version of Debian that the VM is based on. The whole Dekiwiki image is held together by sticky-tape, but that post is really helpful in getting Etch upgraded to Wheezy.

Here are a couple additional things I’ve had to do (please, please look at Jedd’s post above - he covers a whole lot that I’m not covering here):

Upgrading Etch -> Lenny:

Follow the guide at Jeddi.org above to get to Lenny. I can’t remember much failing for that upgrade step, but I remember something happened.

From Lenny -> Squeeze:

DNS Breaks. You’ll notice that SSH takes aaaages to log into, and other things will start being slow and complaining.

Resolve this by editing the ‘hosts’ line in /etc/nsswitch.conf:

hosts: files dns

Originally this reads hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4

Note I’m not actually sure this happened between Lenny and Squeeze; it may have been from Squeeze to Wheezy. *shrug*

From Squeeze -> Wheezy:

MySQL breaks. When visiting the wiki, you’ll be informed that it couldn’t access the backend, and that the API might be warming up. However, you’ll notice that the MySQL service fails to start. When you run ‘mysqld’ from command line, you’ll see it’s missing a folder.

Resolve this by running mkdir /var/lib/mysql-files ; chown mysql:mysql /var/lib/mysql-files

From Wheezy -> Jessie (upgrade-only):

Apache breaks. You’ll notice Apache starts serving just the contents of /var/www. This is because the Apache config file now looks for *.conf in /etc/apache2/sites-enabled; of course, the dekiwiki config file doesn’t have the .conf ending.

Resolve this by running mv /etc/apache2/sites-enabled/dekiwiki /etc/apache2/sites-enabled/dekiwiki.conf

From Wheezy -> Jessie (dist-upgrade):

Mono breaks. If you upgrade mono, Dekiwiki will fall back into unlicensed mode. The licensing engine will no longer be able to interpret license files correctly. You’ll see the error “Server license validation failed. The license signature is not valid.”

Resolve this by holding back the mono packages before you run a dist-upgrade:

apt-mark hold libapache2-mod-mono libmono-2.0-1 libmono-2.0-dev libmono-accessibility2.0-cil libmono-accessibility4.0-cil libmono-bytefx0.7.6.2-cil libmono-c5-1.1-cil libmono-cairo2.0-cil libmono-cairo4.0-cil libmono-cecil-private-cil libmono-codecontracts4.0-cil libmono-compilerservices-symbolwriter4.0-cil libmono-corlib2.0-cil libmono-corlib4.0-cil libmono-cscompmgd8.0-cil libmono-csharp4.0-cil libmono-custommarshalers4.0-cil libmono-data-tds2.0-cil libmono-data-tds4.0-cil libmono-data2.0-cil libmono-db2-1.0-cil libmono-debugger-soft2.0-cil libmono-debugger-soft4.0-cil libmono-dev libmono-firebirdsql1.7-cil libmono-getoptions2.0-cil libmono-http4.0-cil libmono-i18n-cjk4.0-cil libmono-i18n-mideast4.0-cil libmono-i18n-other4.0-cil libmono-i18n-rare4.0-cil libmono-i18n-west2.0-cil libmono-i18n-west4.0-cil libmono-i18n2.0-cil libmono-i18n4.0-all libmono-i18n4.0-cil libmono-ldap2.0-cil libmono-ldap4.0-cil libmono-management2.0-cil libmono-management4.0-cil libmono-messaging-rabbitmq2.0-cil libmono-messaging-rabbitmq4.0-cil libmono-messaging2.0-cil libmono-messaging4.0-cil libmono-microsoft-build-engine4.0-cil libmono-microsoft-build-framework4.0-cil libmono-microsoft-build-tasks-v4.0-4.0-cil libmono-microsoft-build-utilities-v4.0-4.0-cil libmono-microsoft-build2.0-cil libmono-microsoft-csharp4.0-cil libmono-microsoft-visualc10.0-cil libmono-microsoft-web-infrastructure1.0-cil libmono-microsoft8.0-cil libmono-npgsql2.0-cil libmono-npgsql4.0-cil libmono-opensystem-c4.0-cil libmono-oracle2.0-cil libmono-oracle4.0-cil libmono-peapi2.0-cil libmono-peapi4.0-cil libmono-posix2.0-cil libmono-posix4.0-cil libmono-rabbitmq2.0-cil libmono-rabbitmq4.0-cil libmono-relaxng2.0-cil libmono-relaxng4.0-cil libmono-security2.0-cil libmono-security4.0-cil libmono-sharpzip2.6-cil libmono-sharpzip2.84-cil libmono-sharpzip4.84-cil libmono-simd2.0-cil libmono-simd4.0-cil libmono-sqlite2.0-cil libmono-sqlite4.0-cil libmono-system-componentmodel-composition4.0-cil libmono-system-componentmodel-dataannotations4.0-cil libmono-system-configuration-install4.0-cil libmono-system-configuration4.0-cil libmono-system-core4.0-cil libmono-system-data-datasetextensions4.0-cil libmono-system-data-linq2.0-cil libmono-system-data-linq4.0-cil libmono-system-data-services-client4.0-cil libmono-system-data-services4.0-cil libmono-system-data2.0-cil libmono-system-data4.0-cil libmono-system-design4.0-cil libmono-system-drawing-design4.0-cil libmono-system-drawing4.0-cil libmono-system-dynamic4.0-cil libmono-system-enterpriseservices4.0-cil libmono-system-identitymodel-selectors4.0-cil libmono-system-identitymodel4.0-cil libmono-system-ldap2.0-cil libmono-system-ldap4.0-cil libmono-system-management4.0-cil libmono-system-messaging2.0-cil libmono-system-messaging4.0-cil libmono-system-net4.0-cil libmono-system-numerics4.0-cil libmono-system-runtime-caching4.0-cil libmono-system-runtime-durableinstancing4.0-cil libmono-system-runtime-serialization-formatters-soap4.0-cil libmono-system-runtime-serialization4.0-cil libmono-system-runtime2.0-cil libmono-system-runtime4.0-cil libmono-system-security4.0-cil libmono-system-servicemodel-discovery4.0-cil libmono-system-servicemodel-routing4.0-cil libmono-system-servicemodel-web4.0-cil libmono-system-servicemodel4.0-cil libmono-system-serviceprocess4.0-cil libmono-system-transactions4.0-cil libmono-system-web-abstractions4.0-cil libmono-system-web-applicationservices4.0-cil libmono-system-web-dynamicdata4.0-cil libmono-system-web-extensions-design4.0-cil libmono-system-web-extensions4.0-cil libmono-system-web-mvc1.0-cil libmono-system-web-mvc2.0-cil libmono-system-web-routing4.0-cil libmono-system-web-services4.0-cil libmono-system-web2.0-cil libmono-system-web4.0-cil libmono-system-windows-forms-datavisualization4.0-cil libmono-system-windows-forms4.0-cil libmono-system-xaml4.0-cil libmono-system-xml-linq4.0-cil libmono-system-xml4.0-cil libmono-system2.0-cil libmono-system4.0-cil libmono-tasklets2.0-cil libmono-tasklets4.0-cil libmono-wcf3.0-cil libmono-web4.0-cil libmono-webbrowser2.0-cil libmono-webbrowser4.0-cil libmono-webmatrix-data4.0-cil libmono-windowsbase3.0-cil libmono-windowsbase4.0-cil libmono-winforms2.0-cil libmono0 libmono2.0-cil mono-2.0-devel mono-2.0-gac mono-4.0-gac mono-apache-server2 mono-common mono-csharp-shell mono-dmcs mono-gac mono-gmcs mono-mcs mono-runtime mono-utils mono-xbuild

Note: Some of these can certainly be unheld. I haven’t yet gone through and identified exactly which component is breaking on upgrade. If you put in the effort to determine the culprit, please let me know.

That’s all! You should now have Dekiwiki running on Debian Stretch! I’m planning to upgrade it to Debian Buster in the next week or two.

Compiling Heimdall without installing Qt

Wed, 13 Sep 2017 00:00:00 +1200

During my recent Wrecked-phone Saga, I had some trouble flashing my firmware. My phone was broken at the time, so I couldn’t enable ADB Debugging - therefore adb wasn’t an option. OEM Bootlock was on, so fastboot wasn’t an option. My Recovery bootloader was also broken. I thought my phone was bricked.

That’s until I remembered Heimdall.

Heimdall is specifically a tool for working with the partitions on Samsung phones - other visitors need not apply. It works a treat for my Samsung Galaxy S5 (kltedv) though.

Heimdall, however, is old. The version in the Debian Sid repository is 1.4.1, which dates back to 2015. And when we have old versions, we also have incompatibilites! Such as Heimdall 1.4.1’s incompatibility with newer versions of Samsung devices … such as my precious SGS5.

Happily, Heimdall is open source, so I was able to download and compile a new version that overcomes the problem. And that’s how I found myself staring at the screen and looking at this:

That is Yuck.bat

I really didn’t want to install 55 packages just to build this. The majority of missing packages were required to build the GUI section of the tool, which I didn’t want to use anyway. What if I could … build Heimdall without building the GUI??

And of course, that’s the title of this article, so here’s how to do it!

EDIT 2017-12-19: The stuff below is no longer necessary. For better instructions, have a look at my article about flashing a Samsung G900I back to stock.

Clone the git repository, with git clone https://github.com/Benjamin-Dobell/Heimdall.git
Edit the CMakeLists.txt file: nano Heimdall/CMakeLists.txt

Delete the last four lines of CMakeLists.txt:

 cmake_minimum_required(VERSION 2.8.4)

 set(CMAKE_MODULE_PATH
 ${CMAKE_SOURCE_DIR}/cmake
 ${CMAKE_MODULE_PATH})

 project(Heimdall)

 set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin)

 option(DISABLE_FRONTEND "Disable GUI frontend" OFF)

 add_subdirectory(libpit)
 add_subdirectory(heimdall)

You’re done! Go ahead and build according to the instructions.

Respecting Amavis' "Banned Extensions" setting

Sun, 13 Aug 2017 00:00:00 +1200

I’ve been dabbling a little bit with iRedMail, mostly just to have a play with a mail server, but also to see what’s involved in mail security. iRedMail is a package that pulls together Postfix as an MTA, Dovecot as a POP3 & IMAP server, SOGo for ActiveSync, Roundcube for Webmail, SpamAssassin for spam protection, and ClamAV for virus scanning.

Okay I have no idea why I have to write this, but apparently it’s a thing.

Amavis has a list of banned file extensions. In Debian, they live in /etc/amavis/conf.d/20-debian_defaults, and /etc/amavis/conf.d/50-user, and are set in the $banned_filename_re variable. THIS MAKES PERFECT SENSE.

But of course, there’s always something that doesn’t make sense, and that is the fact that there is a SQL backend (at least in the environment created by iRedMail), and settings in here take precedence over the Amavis config files somehow.

And even more bizarrely, there exists in this SQL environment, a policy setting entitled “banned_files_lover”, which was set to “Y”. I shit you not. My only hope is that this only defaults to “Y” for postmaster.

To fix this, you need to hop into the database, and update the appropriate column in the policy table:

$ mariadb
MariaDB [none]> \u amavis
MariaDB [amavisd]> update policy set banned_files_lover="N";

And now, Amavis will obey your file extension filters!

Thankfully, I found this information at http://www.iredmail.org/forum/topic13147-iredmail-support-amavisd-passed-but-setup-at-ddiscard.html – I never would have found it otherwise!

Making Amavis work with ESET Antivirus

Sat, 12 Aug 2017 00:00:00 +1200

But of course, ClamAV has shown disappointing performance, and it would be really nice to use something more… commercially suitable.

To tie together mail receipt and scanning, iRedMail uses Amavis (strictly speaking, ‘amavisd-new’). Amavis uses ClamAV by default, but it comes with a bunch of configuration blocks to bring together other antivirus applications.

But although amavisd-new is stable and still maintained, some parts of it are really old. In particular, many of these av-scanner config blocks are… uhh… “deprecated”. There’s one particular entry for ESET that is dated 2002 - things have changed a lot in the last fifteen years. *shudder*

So, with the help of some documentation, I managed to piece together a code block that works:

['ESET File Security for Linux',
 ['/opt/eset/esets/sbin/esets_scan','esets_scan'],
 '--subdir --unsafe --unwanted --clean-mode=strict {}',
 [0,10,100],[1,50],
 qr/threat="([^"]+)"/m
],

That would be the end of the story, except I also had to hunt around to find out the right place to put my beautiful codeblock into. It turns out that Debian’s Amavis config structure is quite different to the CentOS config that is most-frequently mentioned in the iRedMail forums. I spent a lot of time playing with /etc/amavis/conf.d/15-av_scanners, and nothing seemed to work. Eventually I found out that Debian features a /etc/amavis/conf.d/50-users file that overwrites the settings from 15-av_scanners. Finally I had progress!

Somewhere around line 154 in /etc/amavis/conf.d/50-users, you’ll find an @av_scanners codeblock. I deleted the ClamAV section in there, and replaced it with the ESET codeblock above. I left the ClamAV settings in the @av_scanners_backup section, because Amavis will fall back to that if ESET fails.

That seems to be all! At least, it works with the EICAR anti-malware test file.

WireGuard - Part Three (Troubleshooting)

Mon, 12 Jun 2017 00:00:00 +1200

This is part of my brief series on WireGuard. I’m pretty enamoured with WireGuard and the way it works, but there were a couple slightly curly bits that I needed to get my head around. This troubleshooting guide is a rough dump of the issues I had, and how I fixed them.

Gotten Stuck?

At this stage, there are actually a few ways that this can go wrong, even though we haven’t done much. Think through all the bits:

Installed WireGuard at both ends
Set up your NAT rule on the remote side
Created a private and public key on each side
Put each public key in the opposite side’s [Peer] config
Put a suitable IP address on each side

Troubleshooting

There are, actually, a myriad of ways this can fail:

Broken routing on the client PC
Broken NAT on the remote router
Broken routing on the remote PC
Broken wg0 configuration on one side
Lack of TCP forwarding on the remote computer
Lack of Proxy ARP on the remote computer

If you can’t ping the remote server yet, don’t panic. Run tcpdump to find out what you’re missing.

On the remote server: tcpdump -i wg0
On your local machine: ping -c1 10.20.40.1

That will tell you whether your packets are reaching the remote server, or if they’re not getting through the tunnel.

If they’re not making it through the tunnel at all, you’ll probably be seeing error messages in the ping. For example:

PING 10.20.10.1 (10.20.10.1) 56(84) bytes of data.
From 10.20.40.2 icmp_seq=1 Destination Host Unreachable
ping: sendmsg: Required key not available

That’s the error message I got when I set AllowedIPs too strictly. Because I was trying to ping something that was routable, but wasn’t within the AllowedIPs range, there was no applicable key for the packet.

Thanks

Huge gratitude to Jason Donenfeld (aka zx2c4) for spending his time not only reading this post, but also for sending me some fantastic feedback! I’d made some bungles in my original post on this topic, and he vastly helped my understanding.

Also, huge gratitude to another Jason (@rendition) who has helped me develop from a junior network admin into a … ‘moderate’ network admin. I’ve learned more in the last year than I ever thought possible. He’s taught me nearly everything I know about managed networking, reviewed this post for me, and is actually the guy who introduced me to WireGuard originally!

WireGuard - Part Two (VPN routing)

Sun, 11 Jun 2017 00:00:00 +1200

This is a continuation of my brief series on the new WireGuard VPN. Part One was about the simple building-blocks to get WireGuard working between two endpoints. Now that we’ve got a couple machines able to ping each other by IP address, we can carry on a bit deeper into the inter-LAN routing stuff.

Extending on from the IP addresses in Part One, instead of JUST connecting to the remote machine, I want to actually have access to everything on the whole 10.20.0.0/16 network; even the non-WireGuard devices. I want it to be like I’m there on-site. The idea here is roughly:

Remote machine: wg0: 10.20.40.1 (behind public IP 163.172.161.0)
Local machine: wg0: 10.20.40.2 (with eth0 on 192.168.88.207)
Machines on remote LAN: 10.20.0.0/16
Remote LAN Watchguard range: 10.20.40.0/16 – note this is within the remote 10.20.0.0/16 range

From my local machine, with minimal interaction, I want to be able to ping something like 10.20.10.30, and get a response.

Happily, now that the two endpoints are talking with each other, there’s really not much that has to happen to get things working the way I want.

Remote Config

Many Linux distros disable IPv4 packet forwarding by default. But in this case, we very much want this enabled. Edit your /etc/sysctl.conf file, and look for the line: net.ipv4.ip_forward, and set the value to 1. You may need to add this line manually, or uncomment it.
You’ll also need to enable ProxyARP. This is another kernel setting that’s usually disabled, so edit /etc/sysctl.conf, and add the following line: net.ipv4.conf.all.proxy_arp = 1.
Reload your kernel settings. A reboot will do the trick, or you could just run sudo sysctl -p /etc/sysctl.conf.

Starting WireGuard on Boot

There are a variety options to do this, and if you’re a seasoned sysadmin you probably already have a favourite way.

One option is to add post-up wg-quick up wg0 to the tail of your eth0 block (or appropriate interface) in /etc/network/interfaces.

A second option (for systemd users) is to simply run systemctl enable wg-quick@wg0. This will tell systemd to bring the wg0 interface up once a network connection is established.

I only do this on the remote machine, because I don’t want my local machine to be forever spinning up it’s WireGuard connection; however I always want the remote machine listening for my packets. That said, there’s probably no reason to avoid this; I just haven’t done it yet.

Local Config

Here’s the really cool bit. Since your remote machine is now set up to perform IPv4 forwaring and ProxyARP, the only thing you need to change on the local side is a single number. Or rather, a couple numbers.

In your /etc/WireGuard/wg0.conf file, just expand the network range of your interface to include the entire remote LAN:
```
 [Interface]
 ...
 Address = 10.20.40.2/16

 [Peer]
 ...
 AllowedIPs = 10.20.0.0/16
```
Now reload your WireGuard config, either by rebooting, or running sudo wg-quick down wg0 && sudo wg-quick up wg0.

That’s all! When you make the above changes, wg-quick will modify your routing table so that any IP within 10.20.x.y will be sent over the wg0 interface. The remote host will dutifully forward the packet out into the rest of the network, and ProxyARP will take care of the rest:

$ ping -c1 10.20.10.31
PING 10.20.10.31 (10.20.10.31) 56(84) bytes of data.
64 bytes from 10.20.10.31: icmp_seq=1 ttl=63 time=91.8 ms

Troubleshooting

Okay so by now there’s a really good chance that you’ve bumped into trouble. I’ve started on a basic WireGuard troubleshooting guide, but it only covers the issues that I bumped into. If you’re still stuck after reading through that, ask a question on the WireGuard Mailing List, or reach out via IRC (#WireGuard on Freenode) - this is all on the WireGuard website.

Wrap-up

That should be all! When you reboot the remote server (I specify reboot, because you want to test that your solution can survive an outage at the remote site), you should be able to ping other things in the remote LAN without any additional interaction. I still manually run wg-quick up wg0 on my local laptop, because I don’t want to be connected remotely all the time.

Thanks

If you do end up using WireGuard, go forth and donate! Seriously, at very least, send Jason the cost of a local cup of coffee or a beer for his efforts.

WireGuard - Part One (Installation)

Sat, 03 Jun 2017 00:00:00 +1200

WireGuard is the most excellent VPN stack around. It’s really fast, the concept of Cryptokey Routing is awesome, and I love the speed and simplicity benefits that come from opionionated cryptography. The protocol is so simple - expressed in a mere 4k lines of code - that it’s auditable by anyone.

But.

With my initial naive approach, I found myself using HTTPS, over ports forwarded over SSH tunnels, connected over WireGuard. Although it was straightforward to get WireGuard working between two endpoints, I ended up in nested-crypto hell.

So, this brief series is about making WireGuard work as a VPN onto a different site. We’ll start by making it work between two endpoints, and we’ll go forward from there. My end goal is to have access to all the resources on a remote site, just by running wg-quick up wg0.

The endpoints I’m going to set up are at:

Remote machine: wg0: 10.20.40.1 (behind public IP 163.172.161.0)
Local machine: wg0: 10.20.40.2

NAT Setup

Chances are, your remote endpoint is behind a firewall of some sort. Pick a high port, and configure your firewall to forward UDP packets on that port through to your remote WireGuard endpoint. You don’t need to do this on your local side, because reply traffic from the remote side will generally be handled by the stateful session part of your firewall.

Many routers and firewalls offer port address translation (also known as PAT) as part of port forwarding or NAT. This is when a packet hits the firewall on (say) port 57432, and the firewall puts it on the LAN to port 22, for example. You don’t want this. I think it’s possible with WireGuard, but it adds complexity without benefit.

Config - Remote Site

Install WireGuard

Generate your keys. The following will create a public key and a stub config in /etc/WireGuard/:

 cd /etc/WireGuard
 umask 077
 printf "[Interface]\nPrivateKey = " > wg0.conf
 wg genkey | tee -a wg0.conf | wg pubkey > publickey

Edit your config to match:

 [Interface]
 PrivateKey = WhAt3v3R= (this is the private key generated on this machine)
 ListenPort = 12345 (this is the UDP port you've forwarded from your firewall)
 Address = 10.20.40.1/24 (this will be the IP given to the wg0 interface)

 [Peer]
 PublicKey = (leave this blank for now; you'll paste in your local public key here soon)
 AllowedIPs = 10.20.40.0/24 (this is the range of WireGuard IP addresses that this Peer's key can be used from)

That should be all you need to configure on the remote side for now. Save your wg0.conf file, and bring the interface up:
```
 wg-quick up wg0
 ping -c1 10.20.40.1
```
You should see a bunch of actions performed by wg-quick, and a reply packet from your ping. Now onto the local side.

Config - Local Machine

Install WireGuard and generate your keys, as per the first two steps above.

Edit your configuration again:

 [Interface]
 PrivateKey = WhAt3v3R+PaRt-tw0= (this is the private key generated on this machine)
 ListenPort = 12345 (this is the UDP port again; I don't think they have to be the same)
 Address = 10.20.40.2/24 (note - different IP address, but in the same range)

 [Peer]
 PublicKey = ?????????? (copy the public key from the REMOTE server here)
 Endpoint = 163.172.161.0:12345 (enter the PUBLIC IP address of the remote site, plus the forwarded port)
 AllowedIPs = 10.20.40.0/24 (specifying that packets using this key must come from within 10.20.40.x)

Now that you’ve got a local public key, take a second to paste it into the Peer public key section on the remote server.
Once you’re finished you should be able to bring the interface up and ping it:
```
 wg-quick up wg0
 ping -c1 10.20.40.2
```
… and you should even be able to ping the remote server as well:
```
 ping -c1 10.20.40.1
```

Gotten Stuck?

At this stage, there are actually a few ways that this can go wrong, even though we haven’t done much. Here’s a quick summary of everything we’ve done:

Installed WireGuard at both ends
Set up your NAT rule on the remote side
Created a private and public key on each side
Put each public key in the opposite side’s [Peer] config
Put a suitable IP address on each side

If you’ve nailed each of those and you’re still having trouble, you can have a quick look at the brief Troubleshooting guide I’ve put together as Part Three.

Onto Part Two

That should be all you need to get WireGuard working between two machines on two different sites. So far we haven’t done anything either interesting or uncommon - this is all the basic stuff you’ll find on the WireGuard Quick Start page, although expressed slightly differently. The next article will be a bit more about intra-site routing.

Thanks

If you do end up using WireGuard, go forth and donate! Seriously, at very least, send Jason the cost of a local cup of coffee or a beer for his efforts.

Making Lektor work with grsecurity

Sat, 29 Oct 2016 00:00:00 +1300

I started using grsecurity on my servers in 2015, and there’s always a bit of tuning required.

I was recently playing with Lektor (before I swapped to Pelican), and I had a bit of trouble with my grsec kernel. In particular, Lektor and Pelican are both run within a virtualenv Python environment, and grsec eats it like popcorn in two different ways:

TPE (Trusted Path Execution) throws a wobbly:

[253241.370019] grsec: From {ssh-origin-ip}: denied untrusted exec (due to file in world-writable directory) of /tmp/#50 by /usr/local/lib/lektor/bin/lektor[lektor:60593] uid/euid:1000/1000 gid/egid:1000/1000, parent /bin/bash[bash:60581] uid/euid:1000/1000 gid/egid:1000/1000

This block occurs because the virtualenv violates grsecurity’s Trusted Path Execution protection - the Python executable isn’t under a trusted path, and the user isn’t in the TPE-bypass group.

To resolve this one, add your user to the TPE group. This will allow the user to execute binaries which aren’t in trusted locations:

sudo usermod -aG grsec-tpe $USER

RWX protection. This is a common problem with Python-based apps, even though Python has an exception (Edit 2020-04-12 - Issue 6 from https://github.com/thestinger/paxd/ has been deleted) by default.

[ 207.534876] grsec: From {ssh-source-ip}: denied RWX mmap of <anonymous mapping> by /usr/local/lib/lektor/bin/lektor[lektor:534] uid/euid:1000/1000 gid/egid:1000/1000, parent /bin/bash[bash:477] uid/euid:1000/1000 gid/egid:1000/1000

First, create your PaX ELF headers, and then disable grsec’s MEMPROTECT extension on the included python2 binary, while you enable EMUTRAMP:

sudo paxctl /usr/local/lib/lektor/bin/python2 -c
sudo paxctl /usr/local/lib/lektor/bin/python2 -Em

Result!

Broken log times in APC's PCNS Appliance 4.1

Thu, 27 Oct 2016 00:00:00 +1300

Schneider Electric’s PowerChute Network Shutdown is a piece of software which communicates with your local UPS, and initiates system shutdown if the UPS battery is unable to continue providing power. This helps to preserve file integrity in the event of a prolonged power failure.

Previously, you had to install the PCNS client separately on each virtual machine. Since then though, APC have released a PCNS VMware Appliance which is installed directly into vCenter, and initiates shutdown on all the VMware guests through a single Virtual Machine. This is a much tidier model, so we’ve recently embarked on migrating to the PCNS Appliance.

After installing the PCNS 4.1 appliance and getting everything working, I discovered that the time on all the log files was wrong by a significant margin. I can’t remember exactly what the margin was, it could have been say 8 hours or something.

I did a bit of research and eventually I discovered a spectaularly old question on the APC forums, dating all the way back to 2011:

“I have installed PCNS 2.2.4 into the vMA of an ESXi 4.1 server. It properly communicates with the associated NMC and performs correctly. However, it reports all the activities in Eastern Standard Time. I don’t know where it’s picking up that time zone and can’t find anywhere to change it. In the vSphere client the time shows up correctly. Checking date inside the vMA from the CLI also shows the correct time and timezone. The date and time setting in the NMC is correct and is set to synch with NTP with correct time zone.”
From: http://forums.apc.com/spaces/7/ups-management-devices-powerchute-software/forums/general/6809/pcns-2-2-4-esxi-annoyance

The answers include a broken link to an APC web page about it, but fortunately the poster copied the original post detail, which showed me where to look:

“PCNS uses a standard java function to retrieve the current date from the system when writing events to the log. The Problem on the VIMA is that this function always returns the US date format (PDT).
This is an issue with Java http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6456628
You can correct the time issue on VIMA by modifying /etc/sysconfig/clock file.
+/etc/sysconfig/clock contains a line ZONE=“America/Los_Angeles”+
You need to edit the ZONE= to match your region such as ZONE=“America/New_York”

Sounds great! But when I looked in /etc/sysconfig/clock, I discovered my time zone was already set to “Pacific/Auckland”.

However, it also contained the value “UTC = False”. I changed this to “UTC = True”, and et voilà! The problem is fixed!

Updating the modem and radio firmware on a Samsung Galaxy 5 (G900I)

Thu, 27 Oct 2016 00:00:00 +1300

A couple months ago, Check Point revealed their discovery of the Quadrooter vulnerability affecting the Qualcomm chipsets in oodles of Android phones.

I use –CyanogenMod– LineageOS on a Samsung Galaxy S5, so thankfully I received patches for three of the four vulnerabilities in only a few days.

However, that last vulnerability was part of a proprietary binary blob for controlling the Qualcomm LTE chipset, only patchable by Samsung themselves. It took a while for the Samsung updates to roll out, and then I got distracted for a while, but I finally downloaded my firmware. The package I’ve used is “G900IDVU1CPH3”, which seems to be compatible at least with the three NZ-based carriers, as well as all the Australian carriers.

There are a couple little issues with the firmware upgrade, easily fixed if you know how. I’m going to assume you know exactly how to do everything without help. I’m sure you’re pleased.

In particular, the version of Heimdall in the Debian Sid repositories (1.4.1-2) doesn’t quite work with the SGS5. It appears other people have had similar problems.

The beginning of fix for me came from turboyz on Github, at the bottom of this post: (Edit 2020-04-12: GitHub Issue 348 from https://github.com/Benjamin-Dobell/Heimdall/ has been deleted); however, he’s manually made a couple changes to BridgeManager.cpp, which appear to be no longer necessary. We can simply build it from source:

sudo apt-get install build-essential cmake zlib1g-dev libusb-1.0-0-dev git
git clone https://github.com/Benjamin-Dobell/Heimdall.git
cd Heimdall

# OPTIONAL: Remove the `if(NOT DISABLE_FRONTEND)` codeblock at the end of Heimdall/CMakeLists.txt
# I believe this just saves time compiling the GUI

mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make
cd bin

Now you’ve got a nice new Heimdall installation, with the approproate updates to allow it to talk nicely to the new Samsung bootloader.

I copied my new baseband files into the Heimdall/build/bin folder, just to get them all together. The last step is to flash your NON-HLOS.bin and modem.bin files:

./heimdall flash --APNHLOS NON-HLOS.bin --MODEM modem.bin

I happen to have an issue on my phone currently where, even though the modem and LTE drivers have both been successfully installed, and even thought the Quadrooter tester is no longer showing any vulnerabilities, my Baseband version on my phone is still shown as the old one. I haven’t been able to figure out why yet, but I don’t think I’ll worry about it for now. There have been reports that you should first do this with –no-reboot, wait for the update to complete, then pull your battery, then when you power back on, immediately go back into download mode and re-flash. Apparently this helps update the recognised Baseband version, but to be honest it didn’t help me.

If you’re completely 100% lucky and absolutely nothing goes wrong, you win! You should now have the most recent Samsung modem and LTE chipset drivers available for your phone… without having to roll back to the stock Samsung firmware.

Incidentally, if you want to save gigabytes and time downloading said firmware, and if your phone is precisely a Samsung Galaxy S5 (G900I), and if you definitely want firmware G900IDVU1CPH3, I’ve extracted the important files (modem.bin and NON-HLOS.bin) and stored them here:

https://www.ericlight.com/files/G900IDVU1CPH3_modem_LTE.zip
SHA256 hash = a2ab13063583f6e83a3c2d8b79521a59ba103dda30a24bddf9248dd5a25bff3c

I promise I haven’t intentionally backdoored them, but I make no warranties of any sort. They might just be pictures of my foot.

Good luck!

Using ECDSA certificates with Let's Encrypt

Thu, 27 Oct 2016 00:00:00 +1300

Let’s Encrypt’s Certbot will generate an RSA key by default. But we want to step into the new and exciting world of elliptic curve cryptography! Unfortunately Certbot doesn’t really roll that way, so there are a couple hoops to jump through first.

A word of caution: this post has been written in retrospect, some time after I actually got ECDSA working. That means there are bound to be squiggly little steps that I’ve missed, and I certainly should have provided screenshots or snippets that I’ve missed. Sorry.

This post assumes you’ve already installed Certbot. I had a working regular certificate from Certbot before I changed to ECDSA, so if you have problems following this from scratch, I do recommend trying that first.

For future reference, I’m running the latest certbot available in Debian unstable, which is version 0.8.1-3.

I got most of this information from Scott Helme’s website, which has been awesome.

In brief:

Generate yourself an ECDSA private key:

openssl ecparam -genkey -name secp384r1 | openssl ec -out ec.key

You can change the curve that you use, if you feel a bit wiggly about the controversy around the NSA & NIST degrading the quality of the curves. I don’t feel particularly wiggly about that, myself.

Generate a Certificate Signing Request (CSR) with your shiny new key:

openssl req -new -sha256 -key ec.key -nodes -out ec.csr -outform pem

That will give you ec.key (your private key), and ec.csr (your certificate signing request). Time to get Let’s Encrypt involved.

Create your certificate:

certbot certonly -w /var/www/html/ -d {your_domain} –email “{your_email}” –csr ./ec.csr –agree-tos

If everything goes perfectly, that should leave you with a new shiny set of certificates – quite possibly named something clumsy like 0000-cert.pem and 0001-fullchain.pem, or similar. Throw those into your nginx config and give it a test to see if it’s working.

Schedule your certificate renewals:

I had particular trouble with the renewal process of ECDSA certificates, because certbot renew isn’t compatible with custom CSR’s. You need to run certbot certonly to pass the –csr argument, and then you need to deal with the output yourself.

Even more irksome, the certonly function will fail if you ask it to renew certificates which already exist:

An unexpected error occurred:
OSError: [Errno 17] File exists: '/etc/letsencrypt/live/{your_domain}/cert_ecdsa.pem'
Please see the logfiles in /var/log/letsencrypt for more details.

There doesn’t seem to be any way to tell certbot to overwrite the old certificates automatically, so I created a /etc/letsencrypt/temp folder, and wrote up a really yuck cron job for it. If I were working on a production system I’d do something better, but this works for my lowly domain:

# Recreate certs under /etc/letsencrypt/temp
30 2 24 * * certbot certonly -w /var/www/html/ -d {your_domain} --email "{your_email}" --csr /path/to/your/ec.csr --agree-tos --non-interactive --webroot --cert-path /etc/letsencrypt/temp/cert_ecdsa.pem --fullchain-path /etc/letsencrypt/temp/fullchain_ecdsa.pem

# Backup current certs to home dir
31 2 24 * * mv /etc/letsencrypt/live/{your_domain}/cert_ecdsa.pem ~/certbackup/
31 2 24 * * mv /etc/letsencrypt/live/{your_domain}/fullchain_ecdsa.pem ~/certbackup/

# Move new certs to live folder
32 2 24 * * mv /etc/letsencrypt/temp/* /etc/letsencrypt/live/{your_domain}/

# Restart nginx
33 2 24 * * service nginx restart

And that’s all! It seems to be working so far, but I’m sure something is going to fail at some point – maybe I’ll hit my request limit, or the Let’s Encrypt service will be down, and I’ll lose my certs. If it happens too often I’ll come up with a nicer cronjob and update this post.