HN Reader

Are we decentralized yet?

490

279

11 days agoby Bogdanp

TIL about the Herfindahl–Hirschman Index and I wanted to test it with a weird corner-case that I remember.

At one point in the late 1980's Microsoft had a GREATER than 100% market share of the Macintosh spreadsheet market.

How is this possible?

Market share (for a given period) is the participant's sales in the market divided by total sales. It just so happened that Lotus had more returns than sales of their failed spreadsheet, Lotus Jazz. So Lotus, had a negative market share and Microsoft had more sales of Excel than total sales in the market, resulting in a greater than 100% market share.

I don't remember the exact numbers and I believe there was at least one other competitor in the study. But let's just say the numbers were:

Microsoft: 102% Lotus: -2%

In that case the Herfindahl–Hirschman Index would be 102^2 + (-2)^2 = 10404 + 4 = 10408.

So, in this pathological case it is possible for the HHI to exceed 10,000.

Edited: Added (for a given period) above, for clarity.

11 days agoby msgilligan

Neat! I'm not surprised at the findings here. BlueSky (for the average user) is pretty much a drop in replacement for Twitter.

Despite the smaller total numbers in Mastadon, it's great to see that the ecosystem seems to be successfully avoiding centralization like we've seen in the AT-Proto ecosystem.

I suspect that the cost of running AT proto servers/relays is prohibitive for smaller players compared to a Mastadon server selectively syndicating with a few peers, but I say this with only a vague understanding of the internals of both of these ecosystems.

11 days agoby d4mi3n

So we are not decentralized. Git was a good attempt, but it kind of got centralized around GitHub, GitLab, and other variants. BitTorrent was decentralized, except tracker sites were the natural centralization points. Bitcoin was also decentralized, but still had Coinbase and other sites. Even SMTP is de facto centralized due to the spam problem.

11 days agoby bjourne

We're more decentralized in fedi but we're also not really consistent either. Which I think is the number one gripe for users who manage to get into the fedi.

I don't mind, I still think it's a huge leap forward, but it's important to set realistic expectations.

11 days agoby INTPenis

I'd like to see Nostr added to this, since userbase concentration is always the thing they levy against the fedi model. It'd be a little weird to adapt because user identities don't live on single relays.

11 days agoby treyd

Decentralized has a problem. Its premise is that anyone can set up and run a node, cheaply.

If the decentralized network allows for some kind of targeted broadcasting, it becomes attractive for spamming (e.g: email).

If the decentralized network allows for concentration of responses on something, it becomes a potential tool for a DDoS attack (e.g.: DNS amplification).

So running a node should be somehow expensive, but the expense should be written off if the receivers of the message endorse it, by a one-time action, or automatically by subscribing. An initial credit would allow to establish an audience.

It looks like a perfect use case for a cryprocurrency of sorts %) But this means expensive coin generation, and distribution of the huge ledger across all nodes. That could be delegated to some specialized nodes, but here comes centralization again!

11 days agoby nine_k

How would one measure old school federated contexts like IRC and NNTP in this way? I wonder would they would fare.

11 days agoby jcims

Keep the needle pointing north. Towards the center of that dial.

Too decentralized, and you can't find anything. Nobody uses it.

Too centralized, and censorship takes over. Nobody can speak freely.

11 days agoby Animats

One of the nice things about mastodon is that the total lack of centralization and the absence of the dreaded algorithm means you don't have reputation farmer accounts posting bloviating nonsense the whole time. Its really great there.

Mastodon is social like a quiet pub. Twitter and Bluesky are social like a crowd at a concert.

11 days agoby Devasta

"Decentralization" isn't the end-goal so measuring it here isn't all that meaningful. Personally, I care about:

1. How hard is it to censor the network.

2. How hard would it be for some major player to enshittify the network.

Furthermore, while the fediverse has a single axis for decentralization, BlueSky has 3: number of "big index servers", number of PDSs, number of domain names (how many people own their handle):

1. Increasing the number of PDSs doesn't make it harder to censor the network when everyone still uses the same big index node.

2. BlueSky's primary defense against enshittification is user account portability. I'd love to see metrics on how many users have their own domain names. Having many PDSs is also a good defense here because it reduces the impact of BlueSky (the company) shutting off the firehose, but I still think account portability is the primary defense here.

11 days agoby stebalien

HHI is a pretty interesting metric. It’s calculated by taking each firm’s market share, squaring it, and summing across all firms.

This gives the probability that two randomly chosen customers belong to the same firm.

In one micro models of oligopoly, Cournot competition, it lines up directly with the markup firms can sustain.

Outside of theory, it’s an intuitive way to average together the market power of all firms, with increases in market share for bigger players being weighted more heavily.

10 days agoby oli5679

The universities need to get together and develop their own open-source search engine as part of an ongoing research project. It should be hosted in a distributed fashion from the universities themselves. They have the expertise and the resources these days to do it. And much of the high quality content on the public web originates from the universities anyway. It will be like the Library of Alexandria and not subject to censorship.

11 days agoby espeed

It's so funny watching Bluesky fans defend their almost entirely centralized platform as decentralized just because it has a spec

10 days agoby qwm

The honest truth is that:

Nobody outside tech cares about decentralization or federation.

At some point, everything converges to centralization.

No amount of Mastodon servers or any fediverse self hosted instances spun up will change that.

There is a reason that mastodon.social is the biggest instance and that they couldn't close registrations to promote decentralization.

Hell, I would even say that threads is the biggest mastodon instance.

11 days agoby colesantiago

Including Threads with Fediverse might have an interesting impact.

11 days agoby jeffgreco

> This page measures the concentration of the Fediverse and the Atmosphere according to the Herfindahl–Hirschman Index (HHI), an indicator from economics used to measure competition between firms in an industry. Mathematically, HHI is the sum of the squares of market shares of all servers.

I had not heard of this metric before - it’s neat and simple to understand. If you scaled it down to 0-100 (by dividing by 100), I think it would make the numbers more immediately understandable. I’d even consider inverting it (so 0 = centralized and 100 = decentralized), since the website title implies measuring progress ‘towards’ decentralization.

11 days agoby blintz

Mastodon has been great so far

11 days agoby jarbus

It would be great to add nostr, but nostr doesn't really match this model. Nostr doesn't need a single server to hold your identity, your app connects to many "relays" at the same time.

11 days agoby nout

https://atproto.wiki/en/working-groups/indiesky/stack

multiple different levels of independence to be had in the atmosphere, so it's not directly comparable (doesn't help atmosphere's case though)

i personally am more excited about ATproto as a protocol, and hope https://freeourfeeds.com/ et al can pull it off

11 days agoby throawayonthe

So why not just wordpress and rss?

11 days agoby daft_pink

Wonder how one could calculate email concentration.

11 days agoby BallsInIt

Matrix needs to also be included in this.

Matrix is fancier than mastodon, but more decentralized than AT Proto.

11 days agoby Ericson2314

Personal data detached from the rest is not measured, so I think it’s not realistic to try to determine whether everyone is decentralized. Use of a central decentralized app like Bluesky is not a good approximation imo.

10 days agoby wsdnday

Can someone explain how Fediverse instances or any decentralized platforms deals with most obvious issues of decentralization? Like if I join some node then one day it's offline and all my data lost. I mean surely instances usually don't disappear without notice but it still a totally possible thing. Or what about those times when entire instances get involved in dramas and end up defederating? I don't want to lose my connection with potential friends from such instance just because of something like that. I even remember some time ago when there were news about Threads being federated some people deliberately gathered "signs" across lots of instances to collectively defederate from Threads and it's just ridiculous to me that this even happened. What all this talk about fighting censorship while actively engaging in constant "self-censorship" making own echo-chamber as tight as possible?

11 days agoby qustrolabe

mastodon is the equivalent of bsd code, but for user generated content.

Microsoft and apple would just reach for bsd tcp stack etc. today bluesky just "federate" to mastodon and instantly had millions of active users, until they don't need to federate anymore, when they will blame the other instances to not migrating to some crazy protocol or something, never they will just "shut the door".

11 days agoby 1oooqooq

would be awesome if they added nostr to the list, too.

11 days agoby hugs

I think this compares AT proto PDSes to Fediverse instances. In many ways this actually underplays just how grossly centralized AT proto currently is, since some of the components are 100% centralized still. (Whereas Fediverse instances, all downsides considered, are at least self-contained fully-independent instances.)

11 days agoby jchw

>Mastodon, Pixelfed, etc.

I would like to know the contents of the "etc." list.

11 days agoby natch

Isn't this always a problem of Marketing + UX?

11 days agoby AndreBaltazar

How do these metrics compare with usenet circa ~1989

11 days agoby casey2

I thought Bluesky was federalized? How is it not?

11 days agoby BaardFigur

Depressing

11 days agoby arthurcolle

Sadly, and fortunately, there is no such thing as "avoiding centralization", the evidence is overwhelming:

== Politics & Sociology (power concentrates in organizations)

- Robert Michels, Political Parties (1911) origin of the "iron law of oligarchy": even democratic groups tend to end up run by a few: https://en.wikipedia.org/wiki/Iron_law_of_oligarchy

- Jo Freeman, "The Tyranny of Structurelessness" (1970/72): leaderless groups develop informal, unaccountable elites unless they make structure explicit: https://en.wikipedia.org/wiki/The_Tyranny_of_Structurelessne...

- Max Weber, bureaucracy & rational-legal authority: why modern societies gravitate to rule-bound, hierarchical administration: https://en.wikipedia.org/wiki/Rational-legal_authority

- James G. March & Herbert A. Simon, Organizations (1958): classics on bounded rationality and why attention/decision bottlenecks yield hierarchy: https://www.amazon.se/-/en/James-G-March/dp/0471567930

== Economics & Political Economy (why markets/platforms centralize)

- Ronald Coase, "The Nature of the Firm" (1937): firms exist (and grow) when internal coordination is cheaper than market exchange: https://onlinelibrary.wiley.com/doi/10.1111/j.1468-0335.1937...

- Oliver Williamson, Markets and Hierarchies (1975): transaction-cost economics: asset specificity & opportunism push activity into hierarchies: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1496220

- W. Brian Arthur, "Increasing Returns and Lock-In" (1989): small early advantages + network effects => path-dependent monopolies: https://www.jstor.org/stable/2234208

- Katz & Shapiro, network effects (1985/1994): compatibility and standards help explain winner-take-most dynamics: https://faculty.haas.berkeley.edu/shapiro/systems.pdf

- Thomas Piketty, Capital in the Twenty-First Century (2014): when r > g, wealth concentrates; proposes progressive wealth taxation https://en.wikipedia.org/wiki/Capital_in_the_Twenty-First_Ce...

== Networks, Complexity & Information (why hubs and hierarchies emerge)

Albert-László Barabási, Linked (2002): preferential attachment makes networks develop hubs (central nodes) http://networksciencebook.com/chapter/5

Herbert A. Simon, "The Architecture of Complexity" (1962): complex systems often become hierarchical because modular hierarchies are easier to evolve and manage https://faculty.sites.iastate.edu/tesfatsi/archive/tesfatsi/...

W. Ross Ashby, "Law of Requisite Variety" (1956): controllers need at least as much "variety" as the environment and it often pushes toward central coordinating mechanisms (or many distributed ones with enough capacity) http://pcp.vub.ac.be/books/AshbyReqVar.pdf

Gilbert & Lynch, proof of the CAP theorem (2002): in distributed computing you can’t have perfect consistency + availability under network partitions and real systems centralize/compromise to cope: https://groups.csail.mit.edu/tds/papers/Gilbert/Brewer2.pdf

Robert K. Merton, "The Matthew Effect" (1968): cumulative advantage: success attracts more success, reinforcing centralization of recognition/resources https://garfield.library.upenn.edu/merton/matthew1.pdf

== State power, legibility & infrastructure (why governments centralize)

James C. Scott, Seeing Like a State (1998): states seek legibility and large projects favor central plans and standardized populations/landscapes https://en.wikipedia.org/wiki/Seeing_Like_a_State (literally anything by Jim Scott -RIP- will be useful)

Tim Wu, The Master Switch (2010): communications industries cycle from openness to centralized "information empires" https://www.amazon.com/Master-Switch-Rise-Information-Empire...

== Technology & platforms (contemporary centralization)

Nick Srnicek, Platform Capitalism (2017): explains how platform business models + data/network effects produce concentration https://www.wiley.com/en-us/Platform+Capitalism-p-9781509504...

== When decentralization can work

Elinor Ostrom, Governing the Commons (1990): shows conditions (clear rules, monitoring, graduated sanctions, polycentric governance) under which decentralized, federated management of shared resources succeed https://www.cambridge.org/core/books/governing-the-commons/A...

----

the literature making a counterpoint is abundant / overwhelming but that feels bleak considering when reading these works, systems thinking )the basis for "la technique") favors centralization

11 days agoby DyslexicAtheist