Fail on Think before you push the button

Happy HW Failure day

Mon, 02 Mar 2026 00:00:00 +0000

The server which hosted this website and a bunch of my data-hoarding and data-intensive operations has decided to fuck off, with roughly this failure

[  134.611236] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 32992
[  134.611245] {1}[Hardware Error]: event severity: fatal
[  134.611261] {1}[Hardware Error]:  Error 0, type: fatal
[  134.611266] {1}[Hardware Error]:   section_type: PCIe error
[  134.611269] {1}[Hardware Error]:   port_type: 4, root port
[  134.611272] {1}[Hardware Error]:   version: 1.0
[  134.611275] {1}[Hardware Error]:   command: 0x0547, status: 0x4010
[  134.611279] {1}[Hardware Error]:   device_id: 0000:00:01.0
[  134.611284] {1}[Hardware Error]:   slot: 0
[  134.611286] {1}[Hardware Error]:   secondary_bus: 0x01
[  134.611289] {1}[Hardware Error]:   vendor_id: 0x8086, device_id: 0x3c02
[  134.611291] {1}[Hardware Error]:   class_code: 060400
[  134.611294] {1}[Hardware Error]:   bridge: secondary_status: 0x0000, control: 0x0003
[  134.611297] {1}[Hardware Error]:   aer_cor_status: 0x00000001, aer_cor_mask: 0x000031c1
[  134.611300] {1}[Hardware Error]:   aer_uncor_status: 0x00000020, aer_uncor_mask: 0x00318000
[  134.611304] {1}[Hardware Error]:   aer_uncor_severity: 0x00067030
[  134.611306] {1}[Hardware Error]:   TLP Header: 00000000 00000000 00000000 00000000
[  134.611312] GHES: Fatal hardware error but panic disabled
[  134.611315] Kernel panic - not syncing: GHES: Fatal hardware error

Backups exist, but not for all crap that I ran there. So I’m on a data diet until we attempt a replacement 2 weeks from now.

Daily WTF: AOOSTAR WTR Pro Power Edition

Wed, 25 Feb 2026 00:00:00 +0000

I want this to be an optimistic happy post, I really do. However it just doesn’t work that way isn’t it?

Given that everything is going to shit quickly, including the hardware prices, I thought I’ll stock up on hard drives and populate my storage unit with newer disks. Namely, factory recertified HAMR Seagate drives, of which I’ve got 2x24T and 2x28T. During the course of this forklift I finally managed to put together a working ATA LPM configuration which does not wreck ata port 3 on Aoostar WTR Pro (I mentioned it in one of the previous posts).

However.

What happens instead is, when I insert all 4 disks, the machine doesn’t start properly. It starts spinning up, but power cuts off. If you allow it to suffer, it’ll restart and spin again about 10 times, eventually for drives to gain enough momentum and spin without power fizzling out. And at this point the machine actually proceeds to boot up and everything is grand. But! If for some reason you spin down all the drives, and then try to spin them up, you lose power and have yourself an outage.

I can’t imagine nobody tried to put 4 HAMR drives into this thing. Anyway, there are 2 options - either it’s the PSU and using 200w one will help, or it’s a 12v rail limit and then I need to throw this thing out and put together a new nas - and now it’s the worst time to do that.

How to prepare for ... what, exactly

Sat, 03 Jan 2026 00:00:00 +0000

In my final post for 2025 I theorised that is was the worst year yet. So the coming year (of which we already had a few days) is quickly doing the “yet” bit, starting with the Special Military Trumperation in which Fort Bragg Cartel actually managed to not completely fuck it up (unlike that time in North Korea).

I actually thought this’ll happen before the end of February. But what does it mean exactly, except for the bragging part that I was right? I could probably bet on it, like this dude who is seemingly clairvoyant because they bought Venezuela war bets right before the invasion? Or was this an alt for Pete Kegsbreath?

I think this year we’ll finally see the effects of the shitshow on the global and US-dependent economy; this is the first year in my career when I am genuinely uneasy (that’s an euphemism for scared shitless) about the prospects of the continued employment - it doesn’t matter if you’re doing a good timely work when the pressure happens, perceived scarcity leads to suboptimal outcomes. After all, AI doesn’t really need to be capable of replacing workers - their bosses need to be convinced that AI is capable of doing so.

And, speaking of convincing, I’m genuinely afraid that comunist party will convince themselves that their импортозамещение of ASML is successful enough, so they can wreck taiwan together with their chip manufacturing, sabotage ASML and fabs elsewhere, and become the world’s sole source of high performance computing.

The US has already surpassed most of the authoritarian countries on authoritarian scale. I don’t see any way for that to reverse. Some people genuinely think that midterm elections will be a factor. These are the same people who told me about institutions, checks and balances, etc.

And to finish this pointless rant - an answer to the question “why do you care so much”?

There are lots of very public people who confidently tell their audience that they know how the world works. What follow is, frankly, absolute garbage. Either it’s some stupid conspiracy theory about jews, or soros, or if we go back in time some rotschilds and lizard people. Or it’s some lopsided ideological bullshit like geopolitics, or evangelical eternal battle between angels and demons. Unfortunately most of these people didn’t actually try to manage even the simplest logistical chain for a factory. Or a construction of a datacenter. It is actually funny to read when somebody tries to explain to me how easily Intel can start fabbing 5nm in the US, like, in a month or so.

Unlike these conspiracy theorists, I don’t know how the world works. I know how parts of it work, and I know some of the connections within and between these parts. And some of these connections depend on the rules, stability, predictability, and free global supply chain. And I am scared shitless of these bulls in china shop accidentally breaking a connection which we can no longer replace.

The worst year

Wed, 31 Dec 2025 00:00:00 +0000

Of course it could be worse. The problem is - there’s always could be worse, but there’s also could be better. It’s hard to operate in hypotheticals but sometimes it is blatantly clear that it could be better, damn it.

I participate in a few chat groups, most of them just people who know each other for a while (20 years or so). And I was surprisingly horrified to see some of these people go full moral nihilism this year, quote “if ICE would offer a 350k salary, I’ll go there”. And no moral qualms against working at certain companies, which names are withheld because they index the internet and make a list of future targets for drones.

My resolution for the next year would be to avoid working for ICE and similar companies.

Hoarding useful stuff

Tue, 30 Dec 2025 00:00:00 +0000

I hope you enjoyed my stupid take yesterday, let’s try if I can make a less stupid one.

When you start trying to do something useful, like DIY electronics, or furniture, or art&craft (with uncle rob), at the beginning of the journey you won’t get proper organization with shelves and boxes and categories (unless you’re made of money, have spare room and someone competent set it up for you). As your hobby progresses, you will likely end in a situation when you have many medium-sized containers with completely unorganized contents - like, one part of each type in each of the containers. And there could be many kinds of parts, so it’ll be completely disorganized mess without any way of finding any of that.

I am saying this for a long time that the most critical application for AI now is a machine which can sort garbage. You know, all that stuff you dump into recycle bin, which then gets to some real people who basically do slave labor and sort that shit out into categories. That is the real problem to solve, not any of this vibe science bullshit (which will not work anyway, but try telling that to people peddling it - it’s like trying to tell a jehova’s witness that no, your bible-themed fanfic doesn’t contain all the answers to everything).

Anyway, once that problem is solved, it should be a short hop to solving functional hoarding described above. Just when you release a part, it gets sorted into a bucket! We can even have dynamically printed gridfinity-style buckets! How cool is that!

Originalism as poor design doc hygiene

Mon, 29 Dec 2025 00:00:00 +0000

I was listening to another pointless politics-adjacent podcast today, and they were discussing this concept that some people would know only through conservative politics lens - originalism. Wikipedia tells us that it’s a legal theory in the United States which bases constitutional, judicial, and statutory interpretation of text on the original understanding at the time of its adoption. Originalism consists of a family of different theories of constitutional interpretation and can refer to original intent or original meaning.

As you’d immediately see this paragraph is also a perfect example of another concept - american exceptionalism. I can assure you, this idiocy about reading the old, outdated text and trying to govern the current modern state is not unique to the US, and it predates US by large margin. But that’s not what I wanted to write about.

In our industry we have this idea and concept of a design document, which is sometimes used to describe the system as it should be implemented, and this is the text which will likely be reviewed and debated upon. It’s a useful concept, even though it gets abused a lot. Unfortunately in most cases design documents don’t include the concept of ongoing maintenance, and, what’s worse, they lack reasoning for choices taken. For example, the design doc could describe the API calls or a data structure chosen to hold some elements, but it would omit the reasons behind that:

As the number of active users of the system is assumed to be below 100000, and the pinned state of a single user doesn’t exceed 10 kilobytes, let’s just hold all state in memory on a single machine, in a map.

Most of the design docs actually don’t have that. They would only contain “the user state is stored in memory, in a map” and that’s it. But if we’re lucky, the software actually gets used and these original assumptions become invalid with time. Now we actually have a problem when we try to maintain the system - do we treat the original design as a gospel and never try changing it? What is going to break if we do? What is going to break if we don’t?

I’ve actually encountered so many bad variations of this. The classic “layers” example would be some sophisticated layer next to the original map, the original map changed to hold pointers or IDs into that second layer, etc. And the change would have its own design doc with one of the main selling point be “we didn’t change the original design!”.

You are supposed to change parts of the original design if they no longer fit for the purpose. But to do that, you need to understand the intent and the assumptions behind each of the design choices. Then you would be able to switch to modern technology, or to simplify the system if something is not needed anymore.

And to understand the intent, you need to have it written down somewhere.

So when you write design documents of any kind, please include reasoning for your choices, and assumptions that reasoning is based upon. Things change.

Daily WTF: USB-C PD Edition

Sun, 28 Dec 2025 00:00:00 +0000

About a year ago I’ve upgraded my home server setup to AOOSTAR WTR PRO. It had its share of problems at the start, which sadly included sata instability on port 3. I had an email conversation with aoostar support, and they even sent me a replacement backplane which did not fix the problem. Eventually they’ve sent a new firmware, which kind of helped to the extent. For lack of the better hypothesis I’m now of the opinion that ATA LPM is to blame.

One thing I haven’t tried then is to run this thing from an USB-C power supply. The documentation claims it’s possible, I briefly tried plugging it into Apple’s USB-C brick (heavy version), but the server would not start properly - it looks like it’s running out of power mid-startup, on repeat. I stopped experimenting and forgot about it.

Now I’ve decided to upgrade my UPS setup, and for that I went with BLUETTI Elite 30 V2 Portable Power Station. It was on sale (still is), there’s some content on the internet on how to integrate it with the rest of the stuff, so I thought why not. And it has the USB-C PD output ports.

You can guess where it’s going.

These USB ports don’t do shit with the server. It loses power 3 seconds into boot and starts again. I tried plugging it through power-z but I don’t see anything wrong, it negotiates for 20V and tries dutifully to pull less than 100W, which is the same it does over AC. But still it doesn’t work.

I’m pretty sure a more lengthy debug session is in order. Maybe it’s because the port is wonky? Or maybe it’ll start working if I eject all the hard drives? Or there’s something wrong with Bluetti’s implementation, although my phones work fine with it.

Bottom line is, I just want things to work properly, I don’t want every fucking interaction to require a debugging session.