Posts on Think before you push the button

Vibe coding in Rust

Sun, 08 Mar 2026 00:00:00 +0000

While my machines are still down I am trying to use claude to convert one of my old projects to Rust.

It was originally written in Go (I think I started it on go 1.2). Well, originally it was in C++, on boost::asio, and it was glorious piece of shit.

I didn’t really have any problems with runlib in Go, except it was a bit awkward around the edges where I needed to lock goroutine to a system thread on windows to prevent win32.GetThreadDesktop from exploding. And on linux I needed to do massive cgo abuse plus this.

I don’t know if it’s possible to do better with Rust but the ability to convert the thing at once is valuable - now I can try focusing on rearranging individual pieces to be simpler and to use more of the features of Rust.

Happy HW Failure day

Mon, 02 Mar 2026 00:00:00 +0000

The server which hosted this website and a bunch of my data-hoarding and data-intensive operations has decided to fuck off, with roughly this failure

[  134.611236] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 32992
[  134.611245] {1}[Hardware Error]: event severity: fatal
[  134.611261] {1}[Hardware Error]:  Error 0, type: fatal
[  134.611266] {1}[Hardware Error]:   section_type: PCIe error
[  134.611269] {1}[Hardware Error]:   port_type: 4, root port
[  134.611272] {1}[Hardware Error]:   version: 1.0
[  134.611275] {1}[Hardware Error]:   command: 0x0547, status: 0x4010
[  134.611279] {1}[Hardware Error]:   device_id: 0000:00:01.0
[  134.611284] {1}[Hardware Error]:   slot: 0
[  134.611286] {1}[Hardware Error]:   secondary_bus: 0x01
[  134.611289] {1}[Hardware Error]:   vendor_id: 0x8086, device_id: 0x3c02
[  134.611291] {1}[Hardware Error]:   class_code: 060400
[  134.611294] {1}[Hardware Error]:   bridge: secondary_status: 0x0000, control: 0x0003
[  134.611297] {1}[Hardware Error]:   aer_cor_status: 0x00000001, aer_cor_mask: 0x000031c1
[  134.611300] {1}[Hardware Error]:   aer_uncor_status: 0x00000020, aer_uncor_mask: 0x00318000
[  134.611304] {1}[Hardware Error]:   aer_uncor_severity: 0x00067030
[  134.611306] {1}[Hardware Error]:   TLP Header: 00000000 00000000 00000000 00000000
[  134.611312] GHES: Fatal hardware error but panic disabled
[  134.611315] Kernel panic - not syncing: GHES: Fatal hardware error

Backups exist, but not for all crap that I ran there. So I’m on a data diet until we attempt a replacement 2 weeks from now.

Daily WTF: AOOSTAR WTR Pro Power Edition

Wed, 25 Feb 2026 00:00:00 +0000

I want this to be an optimistic happy post, I really do. However it just doesn’t work that way isn’t it?

Given that everything is going to shit quickly, including the hardware prices, I thought I’ll stock up on hard drives and populate my storage unit with newer disks. Namely, factory recertified HAMR Seagate drives, of which I’ve got 2x24T and 2x28T. During the course of this forklift I finally managed to put together a working ATA LPM configuration which does not wreck ata port 3 on Aoostar WTR Pro (I mentioned it in one of the previous posts).

However.

What happens instead is, when I insert all 4 disks, the machine doesn’t start properly. It starts spinning up, but power cuts off. If you allow it to suffer, it’ll restart and spin again about 10 times, eventually for drives to gain enough momentum and spin without power fizzling out. And at this point the machine actually proceeds to boot up and everything is grand. But! If for some reason you spin down all the drives, and then try to spin them up, you lose power and have yourself an outage.

I can’t imagine nobody tried to put 4 HAMR drives into this thing. Anyway, there are 2 options - either it’s the PSU and using 200w one will help, or it’s a 12v rail limit and then I need to throw this thing out and put together a new nas - and now it’s the worst time to do that.

RFC9580 and you

Sat, 10 Jan 2026 00:00:00 +0000

I have created my original GPG key more than 20 years ago and in the beginning I have used it extensively, but then kind of didn’t. That said, I have a feeling that I’d need to use public crypto a bit more often in the days ahead. To make it easier, I decided to migrate to Sequoia-PGP, rfc9580, etc.

Well it turns out that while I can generate so-called v6 key (rfc9580), I can’t host it on any of the keyservers, because neither hagrid nor hockeypuck support this shit. Now I’ve stored it in DNS, but that doesn’t work for everyone?

In any case, new key’s fingerprint is 693A7DA72B998E1507AE847E394E12F648187357F023A0D7304F9A6570CCCB69

How to prepare for ... what, exactly

Sat, 03 Jan 2026 00:00:00 +0000

In my final post for 2025 I theorised that is was the worst year yet. So the coming year (of which we already had a few days) is quickly doing the “yet” bit, starting with the Special Military Trumperation in which Fort Bragg Cartel actually managed to not completely fuck it up (unlike that time in North Korea).

I actually thought this’ll happen before the end of February. But what does it mean exactly, except for the bragging part that I was right? I could probably bet on it, like this dude who is seemingly clairvoyant because they bought Venezuela war bets right before the invasion? Or was this an alt for Pete Kegsbreath?

I think this year we’ll finally see the effects of the shitshow on the global and US-dependent economy; this is the first year in my career when I am genuinely uneasy (that’s an euphemism for scared shitless) about the prospects of the continued employment - it doesn’t matter if you’re doing a good timely work when the pressure happens, perceived scarcity leads to suboptimal outcomes. After all, AI doesn’t really need to be capable of replacing workers - their bosses need to be convinced that AI is capable of doing so.

And, speaking of convincing, I’m genuinely afraid that comunist party will convince themselves that their импортозамещение of ASML is successful enough, so they can wreck taiwan together with their chip manufacturing, sabotage ASML and fabs elsewhere, and become the world’s sole source of high performance computing.

The US has already surpassed most of the authoritarian countries on authoritarian scale. I don’t see any way for that to reverse. Some people genuinely think that midterm elections will be a factor. These are the same people who told me about institutions, checks and balances, etc.

And to finish this pointless rant - an answer to the question “why do you care so much”?

There are lots of very public people who confidently tell their audience that they know how the world works. What follow is, frankly, absolute garbage. Either it’s some stupid conspiracy theory about jews, or soros, or if we go back in time some rotschilds and lizard people. Or it’s some lopsided ideological bullshit like geopolitics, or evangelical eternal battle between angels and demons. Unfortunately most of these people didn’t actually try to manage even the simplest logistical chain for a factory. Or a construction of a datacenter. It is actually funny to read when somebody tries to explain to me how easily Intel can start fabbing 5nm in the US, like, in a month or so.

Unlike these conspiracy theorists, I don’t know how the world works. I know how parts of it work, and I know some of the connections within and between these parts. And some of these connections depend on the rules, stability, predictability, and free global supply chain. And I am scared shitless of these bulls in china shop accidentally breaking a connection which we can no longer replace.

The worst year

Wed, 31 Dec 2025 00:00:00 +0000

Of course it could be worse. The problem is - there’s always could be worse, but there’s also could be better. It’s hard to operate in hypotheticals but sometimes it is blatantly clear that it could be better, damn it.

I participate in a few chat groups, most of them just people who know each other for a while (20 years or so). And I was surprisingly horrified to see some of these people go full moral nihilism this year, quote “if ICE would offer a 350k salary, I’ll go there”. And no moral qualms against working at certain companies, which names are withheld because they index the internet and make a list of future targets for drones.

My resolution for the next year would be to avoid working for ICE and similar companies.

Hoarding useful stuff

Tue, 30 Dec 2025 00:00:00 +0000

I hope you enjoyed my stupid take yesterday, let’s try if I can make a less stupid one.

When you start trying to do something useful, like DIY electronics, or furniture, or art&craft (with uncle rob), at the beginning of the journey you won’t get proper organization with shelves and boxes and categories (unless you’re made of money, have spare room and someone competent set it up for you). As your hobby progresses, you will likely end in a situation when you have many medium-sized containers with completely unorganized contents - like, one part of each type in each of the containers. And there could be many kinds of parts, so it’ll be completely disorganized mess without any way of finding any of that.

I am saying this for a long time that the most critical application for AI now is a machine which can sort garbage. You know, all that stuff you dump into recycle bin, which then gets to some real people who basically do slave labor and sort that shit out into categories. That is the real problem to solve, not any of this vibe science bullshit (which will not work anyway, but try telling that to people peddling it - it’s like trying to tell a jehova’s witness that no, your bible-themed fanfic doesn’t contain all the answers to everything).

Anyway, once that problem is solved, it should be a short hop to solving functional hoarding described above. Just when you release a part, it gets sorted into a bucket! We can even have dynamically printed gridfinity-style buckets! How cool is that!

Originalism as poor design doc hygiene

Mon, 29 Dec 2025 00:00:00 +0000

I was listening to another pointless politics-adjacent podcast today, and they were discussing this concept that some people would know only through conservative politics lens - originalism. Wikipedia tells us that it’s a legal theory in the United States which bases constitutional, judicial, and statutory interpretation of text on the original understanding at the time of its adoption. Originalism consists of a family of different theories of constitutional interpretation and can refer to original intent or original meaning.

As you’d immediately see this paragraph is also a perfect example of another concept - american exceptionalism. I can assure you, this idiocy about reading the old, outdated text and trying to govern the current modern state is not unique to the US, and it predates US by large margin. But that’s not what I wanted to write about.

In our industry we have this idea and concept of a design document, which is sometimes used to describe the system as it should be implemented, and this is the text which will likely be reviewed and debated upon. It’s a useful concept, even though it gets abused a lot. Unfortunately in most cases design documents don’t include the concept of ongoing maintenance, and, what’s worse, they lack reasoning for choices taken. For example, the design doc could describe the API calls or a data structure chosen to hold some elements, but it would omit the reasons behind that:

As the number of active users of the system is assumed to be below 100000, and the pinned state of a single user doesn’t exceed 10 kilobytes, let’s just hold all state in memory on a single machine, in a map.

Most of the design docs actually don’t have that. They would only contain “the user state is stored in memory, in a map” and that’s it. But if we’re lucky, the software actually gets used and these original assumptions become invalid with time. Now we actually have a problem when we try to maintain the system - do we treat the original design as a gospel and never try changing it? What is going to break if we do? What is going to break if we don’t?

I’ve actually encountered so many bad variations of this. The classic “layers” example would be some sophisticated layer next to the original map, the original map changed to hold pointers or IDs into that second layer, etc. And the change would have its own design doc with one of the main selling point be “we didn’t change the original design!”.

You are supposed to change parts of the original design if they no longer fit for the purpose. But to do that, you need to understand the intent and the assumptions behind each of the design choices. Then you would be able to switch to modern technology, or to simplify the system if something is not needed anymore.

And to understand the intent, you need to have it written down somewhere.

So when you write design documents of any kind, please include reasoning for your choices, and assumptions that reasoning is based upon. Things change.

Daily WTF: USB-C PD Edition

Sun, 28 Dec 2025 00:00:00 +0000

About a year ago I’ve upgraded my home server setup to AOOSTAR WTR PRO. It had its share of problems at the start, which sadly included sata instability on port 3. I had an email conversation with aoostar support, and they even sent me a replacement backplane which did not fix the problem. Eventually they’ve sent a new firmware, which kind of helped to the extent. For lack of the better hypothesis I’m now of the opinion that ATA LPM is to blame.

One thing I haven’t tried then is to run this thing from an USB-C power supply. The documentation claims it’s possible, I briefly tried plugging it into Apple’s USB-C brick (heavy version), but the server would not start properly - it looks like it’s running out of power mid-startup, on repeat. I stopped experimenting and forgot about it.

Now I’ve decided to upgrade my UPS setup, and for that I went with BLUETTI Elite 30 V2 Portable Power Station. It was on sale (still is), there’s some content on the internet on how to integrate it with the rest of the stuff, so I thought why not. And it has the USB-C PD output ports.

You can guess where it’s going.

These USB ports don’t do shit with the server. It loses power 3 seconds into boot and starts again. I tried plugging it through power-z but I don’t see anything wrong, it negotiates for 20V and tries dutifully to pull less than 100W, which is the same it does over AC. But still it doesn’t work.

I’m pretty sure a more lengthy debug session is in order. Maybe it’s because the port is wonky? Or maybe it’ll start working if I eject all the hard drives? Or there’s something wrong with Bluetti’s implementation, although my phones work fine with it.

Bottom line is, I just want things to work properly, I don’t want every fucking interaction to require a debugging session.

Vibeless coding: modularization and function calls

Sat, 27 Dec 2025 00:00:00 +0000

Sometimes I think our industry is pretty unique for having constant inflow of people reinventing the square wheels all the time. Or, to be a bit more specific, some new fad appears and newcomers flock in just learning this new fad, without bothering to learn basics or fundamentals, or even to figure out when and where this fad is applicable. The most recent and egregious example of that is, of course, vibe coding subreddit which has a ton of people who claim to not know anything about programming, but they are very proficient in vibe coding! In most cases it goes as well as you’d expect, with vibes of a pickup-focused FIDO conference from 1990-s, in which men who don’t know how to not scare a woman away are teaching other men how to not scare a woman away.

That was a clickbait, now that I have your attention, let’s do some boring stuff. Specifically let’s try to figure out how to build reliable distributed systems, using zero AI! You might have noticed how it’s now the norm in our world that systems just go bad, and you need to reboot or reset them, and nobody knows how to fix stuff? Right.

Before we start I highly suggest reading just one book: Philosophy of software design by John Ousterhout.

The need for modularization

As it appears, there’s only one way to write software which you can understand and modify - to make it modular. It’s true because Ousterhout says so! Why are you not believing me now???

Consider this simple mathematical expression:

x / 2

This is what you’d write in most of the modern programming languages, and then it’ll be the job of the programming language to figure out which are the correct sets of instructions for the given type of x, on which CPU hardware, etc. This is the simplest to express (but not simplest internally) form of modularization and it is also the simplest example to show that it is the only way to make programs maintainable by humans. If we don’t have this, then I’d need to write all these different types of divide by hand, and then how I’m supposed to change it or even understand that I’m dividing? I’d had to write a comment somewhere saying what this mess of assembly is doing.

Same goes to the higher forms of modularization as well - which is everything upwards of functions:

`1`	`result, err := CopyFile(ctx, "file:///etc/passwd", "s3://public-bucket")`

In this example we let CopyFile figure out how to handle different URLs - and it’ll probably do it by subclassing URL handler and using interfaces etc.

So it is obvious that we need modularization to make it possible to write things with adequate velocity. We also need correct modularization which hides unnecessary stuff and exposes required stuff, and you can read all about it in a book referenced above. But that is just for interfacing between human and machine.

What if I told you we need modularization for machines as well?

Before we do so, let’s do a refresher on how modularization basically works (on modern hardware, with compiled languages, most of the time).

Local function calls

Modern CPUs, when you look at them from the “consumer” point of view, eat code instructions from memory and execute them. The instructions could be primitive (data copying, math) after which the execution pointer (usually a CPU register) will move to point at the next instruction in line. Or it could be control flow instructions which will modify execution pointer (JUMP or B code), sometimes they can be combined or used in conjunction with conditionals (for example, JNZ - jump if not zero). And, because the only way to do modules is to do function calls, all modern CPUs have an equivalent of a CALL instruction, which saves the address of a (normally) next instruction somewhere and then jumps into destination, so then destination at some point can execute RET and return the control. Now if you don’t know what stack buffer overrun yet, go read about it but please return.

So with all that in mind, in the example with x/2 when, for example, CPU doesn’t support the floating point operations the compiler could have a function fp_div somewhere in standard library and then call it. Or, if x is of a custom type and we’re using c++, the compiler could call our operator/.

But we’re not done yet with this control flow. What is happening now is blocking call - the execution of a linear, imperative program is blocked until the control is returned. In provided example with c++ and division our execution core is a CPU core and the contents of the function are executed on the same core. But it doesn’t have to be - other function can launch calculations on other CPU cores, or sleep for 1 second - from the point of view of the function caller we are blocked until the function returns.

Why does it have to be blocking? Actually it doesn’t, we just defined it poorly. Imagine this code fragment:

1
2
3

  value1 := Calculate("value1")
  value2 := Calculate("value2")
  value3 := Together(value1, value2)

In this example, we don’t need value1 until we start computing value3. We can write a framework or a language which would automatically handle such cases and spawn calculations in parallel in background. But our primitive imperative languages can’t do it by themselves, and even if we have such a framework we need to know what we’re doing.

So, to summarize:

The only thing we need for making software is ability to define functions and call them.
Most primitive way to do that is implemented in hardware on most modern CPUs.
Function calls in imperative program are blocking the execution of a said program, until they return with a result.

WTF is an API

As we learned, it’s only with modular systems and abstractions we can build something great, because we’ll be building it standing on the shoulders of

Strictly speaking, even the bare CPU with no operating system or library support has an API. It consists of the CPU instruction set which you can use, possibly, to write more complex functions, which in turn will be used to write more complex functions. The SOCs we get today usually come with an SDK which includes at least a compiler and a library to control various embedded things in that CPU, without resorting to assembly (functions like “go to deep sleep” or “initialize wifi” for esp32). But, on top of that, the operating system will offer you APIs to read/write files, do things with network, write things on the screen, etc. And your language will have a standard library which will usually plug into these OS APIs. It’s APIs all the way down and it’s pretty neat, until it doesn’t. And when it doesn’t it means we have an error somewhere which we need to handle. And if the error happens during long-running operation, we need to make sure our long-running operations are designed and implemented correctly. And that’s what we are going to cover next - errors and long-running operations.

Personal note organizers, how do they work

Sat, 18 Oct 2025 00:00:00 +0000

I actually have this very thorough post in the works about replacing old Nest doorbell with a new Unifi Doorbell Pro 4. But putting it all together is not easy! I am trying to make it ring a digital chime, reliably and without wi-fi in the path, and nothing works; I think what should work is a relay trigger, but now I need a relay which accepts AC control input.

So while that is not ready, what is the topic anyone (including me) can write on without any preparation and even without knowing shit about? Politics of course.

Not today.

Today I am trying to start using personal data organizers (or whatever this thing is called nowadays) in a way which works? How is this supposed to work at all, if I’m lazy and disorganized? So far I tried:

Logseq then realized the android app is just an apk and it needs full permissions to all files.
Obsidian - I spent a day setting up livesync (not the official one), mostly fighting with couchdb. It kind of works now, but then I found the one below.
Silverbullet which is the current favorite, just because I did set it up quickly.

I have no idea which one I’m going to stick with long term, but currently all my notes are in a silverbullet instance; I’m trying to actively use it for organizing travel, taxes, and of course for shitposting! What I don’t want to do yet is make my entire website a silverbullet, like the silverbullet.md itself.

Cheap Yellow Displays

Thu, 19 Jun 2025 00:00:00 +0000

I’m trying to decorate my house with Cheap Yellow Displays, buying a bunch of different models from aliexpress, with different levels of success. And I’m gonna tell you - why is the hardware so bad? So far I found that on the USB-C variant with capacitive screen (2432s024) usb-c isn’t a real one, it doesn’t work with usb-c power but only with A-to-C power (most likely, lack of proper CC resistors).

Anyway, reason I’m posting this is so the internet (which is full of AI garbage) will contain at least 2 copies of this code:

i2c: # required for touchscreen
  sda: GPIO33
  scl: GPIO32
  scan: true

touchscreen:
  platform: cst816
  # interrupt_pin: 21
  update_interval: 50ms
  reset_pin: GPIO25
  on_touch:
    - logger.log: "Touch event!"
  on_release:
    - logger.log: "Touch release!"
    - if:
        condition: lvgl.is_paused
        then:
          - logger.log: "LVGL resuming"
          - lvgl.resume:
          - lvgl.widget.redraw:
          - light.turn_on: backlight

Yeah, interrupt_pin didn’t work for me, only polling with update_interval. Also, scan:true but it will not show anything on boot, screen is reacting to touch but invisible on scan, how cool is that.

I’ll have a properly sized post about it at some point.

The plot thickens with MAX31865

Sun, 11 May 2025 00:00:00 +0000

My max31865 story is still ongoing. I managed to find a PT100 sensor to pair with my accidentally obtained boards. It worked in testing, but once I hooked up to the power supply I was using for my solar interface board all along, it doesn’t work! It complains about “over/under voltage” and can’t calculate the resistance. I thought it’s because the power supply is weak, so I added a 6300uf capacitor to the 5v line. It didn’t help. I still haven’t figured out what is happening, in the meantime I just powered it from USB-C 5v PSU. At least I have some temperature measurements; also, there’s apparently a PT1000 variant of the breakout somewhere in the mail, destined for my postbox. We’ll see if that one will be twitchy too.

On a completely unrelated note I find it increasingly hard to do anything useful nowadays. I don’t know if that’s some new form of burnout, but I’m trying to find an interesting project to do and can’t. Everything is no fun. I should try what always helps - making a list of things to do.

Reinstalling old machine over the network

Fri, 25 Apr 2025 00:00:00 +0000

I have an old machine in an undisclosed location which I use as one of the offsite archives. Unfortunately there was something with power there, and then I hit one or two bugs in f2fs so it died. No big deal, I asked the person who takes care of it to plug another SSD. But.

The machine in question is HP ProLiant ML110 G6. It has IPMI in it (well, HP’s own iLO to be precise). And it has a java applet which you can use for virtual console, if you nerf your JVM enough so it will allow ALL the unsecure protocols. Within this applet, you can attach a virtual CD rom which then can be used to boot the machine and install it. But unfortunately, fedora’s netinst images are too large for it. The reads are slow but also the applet fails after some time (or after a number of sectors read).

So, how did I solve the problem? With the help of ipxe!

I used ipxe before but mostly to implement the automated software install - as opposed to classic pxelinux, ipxe can be scripted to download things over http which is much better than tftp, especially if you want to time reinstall 100 machines at once.

Ipxe can be made into 1 megabyte ISO image. So that’s what I did - tried to boot into it, then fooled around in ipxe console to confirm that it can see the network card, then wrote a bootstrap script and ran make bin/ipxe.iso EMBED=/home/stingray/script.kpxe.

And this is what was in script.kpxe:

#!ipxe

set net0/ip 192.168.1.13
set net0/netmask 255.255.255.192
set net0/gateway 192.168.1.1
set dns 8.8.8.8
iflinkwait net0
ifstat

# The following is from https://ipxe.org/howto/fedora
# Set source URI
set mirror http://download.fedoraproject.org/pub/fedora/linux/releases/42

# Detect CPU architecture and calculate repository URI
cpuid --ext 29 && set arch x86_64 || set arch i386
set repo ${mirror}/Everything/${arch}/os

# Start installer
kernel ${repo}/images/pxeboot/vmlinuz initrd=initrd.img ip=192.168.1.13::92.168.1.1:26:myhostname:enp30s0:none inst.repo=${repo} inst.rdp inst.rdp.username=INSTALLER inst.rdp.password=FREELUIGI nameserver=8.8.8.8 inst.geoloc=0
initrd ${repo}/images/pxeboot/initrd.img
boot

Of course this would look much simpler if you have working DHCP on the network. I didn’t.

MAX31865 conspiracy

Sun, 20 Apr 2025 00:00:00 +0000

I’ve finally decided to install solare electrical (photovoltaic) system. Unfortunately this means I need to get rid of existing solar heating system. Not a big deal, except I’ve used the data from the controller to track the hot water temperature. So, naturally, the idea was to put together something which can take existing PT1000 leads in.

ESPHome has a component, so naturally I went to aliexpress.com and ordered 2 breakouts which look the part. Unfortunately, these ended up having 430 ohm ref resistor, which means that they can only be used with PT100. Now, it doesn’t seem to be possible to know if the breakout offered on aliexpress has 430 or 4300 ohm resistor in it. So I ether need to find 4300-ohm breakout, or I need to unsolder the resistor and solder another one in place.

Small and Meaningless things for small people

Sun, 09 Mar 2025 22:25:55 +0100

For a while now I’m working (slowly) on a monumental series of articles about function calls, RPC protocols and arrangements, APIs - all in line to present best and worst practices of modern software engineering. I would like to do it faster, but:

Any country, corp, org, or individual that uses Starlink is subject to the whims, wishes, demands and insults of The Owner. Do so at your peril.

[image or embed]
— Ron Filipkowski (@ronfilipkowski.bsky.social) March 9, 2025 at 5:18 PM

That’s the new reality, folks. Apparently we are all small men, exist at the whim of the new sun king. Who fucking knew we will end up here.

Well, nothing matters anymore, so let’s try doing some completely meaningless things instead! For example, systemd service hardening!

systemd-analyze security

Here’s a starting point for y’all. Run systemd-analyze security sshd. Looks funny, isn’t it. All right, to bleach the console run systemd-analyze security irqbalance (all commands assume fedora 41). But for one irqbalance we’ve got a hundred services which say “AAAA UNSAFE”. What’s going on?

Same thing as always - it is hard to add security as an afterthought. Historically, all the services were barely protected - before systemd and selinux, best in class was “run as a different user”. With selinux, it became possible to control all aspects of what the service does - except that almost nobody is capable of doing it, because you need to understand in the greatest detail what the service does and how, and the prevailing culture in our industry is such as the author of the service will rarely bother with that.

Systemd services brought massive quality of life increase in terms of service configuration, but also provided a few options to improve service security, which are less heavy than selinux. Except even this easy mode is too hard for everyone.

Anyway, supposedly I care. With selinux, technically I can run things in permissive and use audit2allow to produce or fine-tune a policy. It’d be nice if similar things existed for systemd? Well, it does, it has an unsurprising name Systemd Hardening Helper and it works! To an extent. There are a bunch of todos and not all hardening options are implemented.

The basic idea is the same as audit2allow - run the service under strace and parse it. That’s it, the UX is reasonably good, and while - as mentioned - not all options are covered, it provides a good starting point for further improvements.

It’s written in Rust, so here’s a chance to learn more of it and improve the security of everyday services. And with all the shitshow going on, we need all the security we can get.

Until the next time, folks. Please try shielding yourselves from orbital bombardment in the meantime.

Rustic vs Restic

Fri, 10 Jan 2025 22:18:06 +0100

As always, I promised more updates and then immediately disappeared. This is a common pattern - or at least it’s something I do regularly. We have this tool at work called “snippets”, and I usually start writing there but then it’s kind of hard to maintain consistent schedule. I should probably write a dedicated rant about this at some point.

Anyway, today it’s going to be short and pointless.

A story of some backups

For a long time I was using borg-backup for my personal backup needs and it was dire. It was painful to manage and work with (I wrote a custom wrapper so it could work over https), the performance was awful. Then restic appeared, and I almost jumped on it full speed. Then I realized that without compression my backups are 5x bigger than they should be.

Anyway, at some point restic finally added compression, so this time I’m definitely going to do it! However, it just so happened that pack size was capped at 16 megabytes or so, and when writing to the filesystem it produces a gazillion of files. Which is not du-friendly, or any fs-friendly at all.

At that time I was experimenting with haystack-type storages (specifically, I wanted to write a library which gives you an append-only haystack with leveldb-based index on the side), so naturally I wrote a rest-server on top of it. And it was great! It just, you know, takes more backups and that’s it. Shiny!

This xmas I decided to shuffle things around, including with the backups (which reached 3T). So I started digging some threads and stumbled upon a post by some dude who was saying that in rustic packsize is a function of the reposize. At first, I thought “huh weird typo”. But then - well, there’s the whole restic rewrite in rust!

So that’s it, folks. I’m moving to rustic. Repacking all my backups in the process. I hope I’ll still have at least some of my data.

Next up: how to replace a backplane in Aoostar WTR Pro (and why).

Comments via Bluesky

Sat, 28 Dec 2024 22:38:47 +0100

In my new year’s “resolution” I have “resolved” to add comments to this site via bluesky. That was definitely fun but only to an extent; next time I shall pledge to use unreal engine 5 for comments or something.

UPDATE: That was a bad idea. While bluesky looks nice as a replacement to xwitter, without direct posting box and styling which I can’t do it doesn’t look good at all as a comment engine.

Anyway, there seems to be no “official” social options for bluesky in any of the themes I tried (but I tried only a few), but there’s a ton of specific recipes on the internet, such as:

This seems to have the most features, but it infects your computer with node_modules, and we all know it doesn’t end well for human civilization.
This just embeds the entire bsky post in your page. If I do that, it’ll be the most good looking part of the website, and I can’t allow that. People who elected to read this must suffer.
This looks simple enough, I’m going to try it here now.
- This doesn’t add an edit box, only “Reply to this post on Bluesky to leave a comment”.
- It looks like shit.
This is supposedly a version of the first option without node_modules. It also looks like shit and that is what I ended up using for the whole day! Then got rid of it.

It appears that everything except embedding the entire post looks like shit. And all of this is read-only - that is, there’s no “enter comment here” box. If you want to leave a comment, go to bsky.app and post there.

And after fiddling with it for a day I’ve decided that UX is terrible. We’ll get back to it again in a few months.

Useful technical content begins now

Sat, 28 Dec 2024 01:35:24 +0100

It’s this time of the year when people are trying to make a summary of achievements and make “resolutions” for the new year. Usually none of that is useful, especially the resolutions part. But it’s definitely entertaining to watch and especially to participate. The part I like the most is when it’s around March, and you look at the list of your resolutions, and you haven’t done a single thing. But I digress.

My resolution for the new year would be to waste less time and finally produce some more useful content. Specifically that would be code and documentation, which other people (including, but not limited to, you, dear reader) can use to make the world less shitty place.

There’s one problem which plays into this whole “resolutions are never done” trope. I think most of the content I can produce is trivial and isn’t worth it. Nobody will learn anything useful from it and nothing will be done with the world, it’ll remain at least as shitty as it is. This mindset really cripples my motivation and as the result - nothing in done.

But in reality this is false. To prove this I’m going to undraft an incomplete post I was sitting on for 2 years about institutional knowledge, which I haven’t finished because I wasn’t able to find a good punchline. The fact is, while I think some of the things I know are trivial, unfortunately most of the people in our industry don’t have a clue, and others (and their users) are suffering because of it. And this is why I’m going to make much more of the effort this year to share my rants with you.

The real problem with this will be not that it’s too trivial, but the opposite. For some of the topics to accept the conclusion you need to be well-versed in the prerequisites, which we also need to cover. This is going to be fun. I shall enable comments in which you’ll be able to tell me how much sense I am making and hopefully that’ll direct me to the most efficient approach. I think I saw the way to do it with bluesky? But first I need to move this page to cloudflare for more flexibility (this is done!), and to protect you against occasional IPv6 hiccups (TODO: write about IPv6 hiccups, troubleshooting, and the “root cause”; also TODO: write about why there’s no such thing as root cause and what is, in my opinion, the shitty state of our industry wrt postmortems).

So in no particular order I shall have a:

Story about replacing my old HP Microserver Gen8 with new Aoostar WTR Pro (not without problems! still ongoing!)
Rant about rootless podman
Story about replacing old ikea tradfri gateway with HomeAssistant + Zigbee2mqtt
Story about Ubiquiti and sad state of home networking.

That’s all for now! Stay safe these days!

Kubernetes at home

Sat, 11 Mar 2023 23:25:00 +0100

I decided to finally try installing kubernetes at home. This is the collection of rants and random thoughts I have about it.

I used k3s - works on 1 host so far, don’t know how much I can scale it? Default settings. I Haven’t tried dual-stack/ipv6 yet. I will soon.
Longhorn as storage. Again, hard to give it meaningful testing with one raspberry pi 4 node. I expect abysmal performance all the way through.
Setting up a private registry for the bootstrap shouldn’t require me to run it in containers. Should I write a self-contained single-binary trivial registry because nobody bothered - it’s either docker-registry, quay, or harbor?
My hello world works but I’m guessing I need to make it docker-aware. At least it needs to manage the local space in mounted volume correctly.

Institutional Knowledge

Sun, 29 Jan 2023 21:40:21 +0100

I was talking to a bunch of friends the other day, and one of them started telling a story how in a startup they’re working at production postgres instance was slow, and there was some deliberation on making it a bit bigger (ordering a bigger, more expensive virtual machine), so this was going on for a while and he asks the other guy - “do you know why it is slow? did you run explain on it?”. So the other guy was confused, turns out they didn’t know what EXPLAIN is, and they didn’t have the correct indexes and once they added the correct index they’ve had a 600x speedup without changing the shape of the VM.

I pointed out this was nothing new, as I did exactly the same with out common acquaintance back in year 2000, where that person was complaining that their database is slow and they need a new server, and I asked if their queries are using the correct indexes (“what is an index”).

Another friend in the call joined the party and told a story of the application which fetches the entire contents of the database and then filters out a few rows that’s needed in the app, and the lead developer (who used to be a CTO of a startup) wasn’t able to speed it up no matter what they did! Except one thing that they didn’t, is to write a correct database query which only fetches the records they need. Again, the concept known at least for past 20 years, but suddenly unavailable.

What all of these cases have in common, you’d ask? They all prove the point that our industry is consistently failing to solve well-known problems and keeps cycling around them in weird self-repeating nonsense loops. It should be basic common sense to use indexes (and, in general, use data structures and access methods adequate to the task at hand). But no, everyone is self-taught and because they were learning their databases building their joke of a website with barely 3 visitors per day, they never needed the performance so they never learned.

A short guide for choosing a local file system

Sun, 29 Jan 2023 01:20:36 +0100

So we have a linux machine and couple of new “drives” (block devices), and we want to put a few (millions of) files on these. Local drives, nothing fancy or distributed - which means normal, regular linux filesystems.

We may or may not have SSDs, and we may or may not have a hardware RAID already.

Why it’s never Ext4?

Pretty much the only place you’d want to use ext4 now is /boot. It is not the best choice anywhere else. For regular flat partitions (if we don’t care about the integrity, or there’s a hardware RAID underneath) please use XFS. For lone small-ish SSDs please use F2FS.

For a long time the best way to handle many spinning disks was a stack of MD+LVM+XFS. Now BTRFS gives you the best in terms of managing it, so you’re able to both add and remove disks, and run periodic scrubs. It is probably not the worst option for handling multiple SSDs in a mirror (but this may not be accurate - XFS + md with a bitmap may be a better choice as it may avoid Copy-on-write amplification).

BTRFS pros and cons

Pros first:

BTRFS stores block/extent/metadata checksums. Traditional MD/RAID in a mirrored mode doesn’t store the checksum, so if there’s an error which results in data bits changed (and it evades the hard disk’s checksumming mechanism itself) it’s impossible to know which of the 2 copies is correct. BTRFS doesn’t have this problem.
You can add and remove disks, one by one, mixing disk sizes and utilizing the maximum of those, with data moved between disks on “what’s occupied and what isn’t” basis.

Cons next:

Copy-on-write nature of btrfs makes it the worst medium for storing databases. Postgresql, mysql, you name it - all of these will produce crazily fragmented files and insane amounts of write amplification.
The performance of raid5-6 configurations is laughably low. Of course, this is true for any implementation of raid5 (also, almost any implementation of raid5 has the write hole).