Show more

We did not know about this beforehand as it is not noted in any docs we could see regarding distro or pg upgrades. We noticed strange things approx. 6 hours after the distro+pg upgrade with pg_upgrade and had ~10 unique index violations in our relatively small Mastodon instance, ~10 across 2 smallish HackMD instances, and ~30-40 across 2 rather lively Matrix Synapse instances. It took us around 3h to get everything sorted and can only _hope_ everything is good now. Please read the linked wiki.

Show thread

Affected upgrades are Ubuntu 18.04 or lower to 18.10 or higher, so especially current LTS upgrades (18.04 to 20.04), Debian 9 or lower to Debian 10, RHEL/CentOS 7 or lower to RHEL/CentOS 8.

Not heeding the linked advices will at some point in the future lead to strange search and ordering results out of your db and will probably cause duplicate entries despite unique indices. Fixing this afterwards involves rebuilding the indexes and somehow fixing any duplicate key error you encounter.

Show thread

Important ASA (Admins Service Announcement) for Mastodon and generally PostgreSQL admins: Due to some changes in glibc some distribution upgrades will cause PostgreSQL text indexes to become corrupted, potentially leading to unique indexes not being correctly enforced and inconsistent application data.


Judging by the number of genuinely positive feedback I have received, the most useful piece of information I have handed to my colleagues in these past 3 weeks, is the Mac shortcut to type . [⌘]+[ctrl]+[spacebar]


These days "DevOps" is loose term, purposefully kept ambiguous because .. well, largely because it pays well to leave it obscure and opaque 🤑

Join me and other folks in the community to take a stroll down memory lane to dig up what DevOps isn't, what it really is, and how it can help you and your colleagues harness its philosophy and methodologies to build better products and, ultimately, provide more value for your customers:

Fairphone and /e/ OS have teamed up to introduce the first privacy conscious and sustainable smartphone on the market. Listen in as we chat to, the founder of /e/ foundation:

Welcome #Fairphone3 to our range of #deGoogled phones!

This is probably the 1st privacy conscious and sustainable phone, fair for the planet and your personal data.
Preorder your /e/-Fairphone 3 today!
@Fairphone @WeAreFairphone #privacy #smartphone #yourdataisyourdata

Very proud to have released the first chapters of „The Rails 6 Way“, the book I‘m writing with Obie and Tom <3 It is a lot of work, but that makes me even happier to have reached this milestone

The best article on io_uring and eBPF I’ve read in a while, about why it has the potential to revolutionize development on Linux.. super interesting!

And we've made it across .. enjoy Mastodon 3.1.3 (and a few $$$ spent less each month!)

Show thread

For some reason I "missed" the latest update a few weeks back .. going to bump us to 3.1.3 in the process of shuffling around some infrastructure in the background. This shouldn't cause any downtime.

It's surreal .. with the pandemic, everything you do right will never be noticed, yet all your mistakes might cost you dearly and everyone will be on the fence ..

Never had I thought to see the day where the job I have been doing for such a long time hits reality as harsh as today 🤔 #sysadminlife

For y'all complaining about Zoom .. Skype for Business has a "Web App" which needs to have a Windows MSI installed to actually work :blobfacepalm:

If you're feeling lost right now, and you want to know what to do. How do we organize now? A comrade in the communist collective I'm in put together this really good introductory text compilation that I HIGHLY recommend you read:

It's only 54 pages, it's not dense, it's very accessible. You can read it all in 1-2 hours and you will feel a lot more prepared afterward. It's worth your time whoever you are, I promise.

me after a shower: *carefully drying myself off in the shower with a towel, open up shower door for a second, stepping on discarded towels and laundry while tip-toeing through the bathroom as to not get everything wet*



Show more

A Mastodon instance running on ThoughtWorks infrastructure for its employees to interact with the Fediverse.