HN Reader

It's also hard when the team actually cares, but there are skills you can learn. Early in my career, I got into solving some of the barriers to software project management (e.g., requirements analysis and otherwise understanding needs, sustainable architecture, work breakdown, estimation, general coordination, implementation technology).

But once you're a bit comfortable with the art and science of those, big new challenges are more about political and environment reality. It comes down to alignment and competence of: workers, internal team leadership, partners/vendors, customers, and investors/execs.

Discussing this is a little awkward, but maybe start with alignment, since most of the competence challenges are rooted in mis-alignments: never developing nor selecting for the skills that alignment would require.

For instance, I had the opportunity to review an RFP put out by a state government for software to run a single state government. The specifications stated that a SINGLE software should be used for running the full administration of all of the departments of the government - including completely disparate things such as HR, CCTV management, AI enabled monitoring of rodents and other animals near/at warehouses, all healthcare facilities, recruitment, emergency response services etc...

ONE SOFTWARE for ALL of these!

There isn't a single company in the world who can write software to monitor rodents, hospital appointment booking, general payroll, etc. And since the integration required was so deep, it would be impossible to use existing best-of-breed software.. and everything has to be written from scratch.

How is such a software project ever going to suceeed?

Mostly the issues are non technical and grounded in a lack of accountability and being too big to fail. A lot of these failures are failing top down. Unrealistic expectations, hand wavy leadership, and then that gets translated into action. Once these big projects get going and are burning big budgets and it's obvious that they aren't working, people get very creative at finding ways to tap into these budgets.

Here in Germany, the airport in Berlin was opened only a few years ago after being stuck in limbo a decade after it was supposed to open and the opening was cancelled only 2 weeks before it was supposed to happen. It was hilarious, they had signs all over town announcing how they were going to shut down the highway so the interior of the old airport could be transported to the new one. I kid you not. They were going to move all the check-in counters and other stuff over and then bang on it for a day or two and then open the airport. Politicians, project leadership, etc. kept insisting it was all fine right up until the moment they could not possibly ignore the fact that there was lots wrong with the airport and that it wasn't going to open. It then took a decade to fix all that. There's a railway station in Stuttgart that is at this point very late in opening. Nuclear plant projects tend to be very late and over budget too.

Government IT projects aren't that different than these. It's a very similar dynamic. Big budgets, decision making is highly political, a lack of accountability, lots of top down pretending it's going to be fine, big budgets and companies looking to tap into those, and a lot of wishful thinking. These are all common ingredients in big project failures.

The software methodology is the least of the challenges these projects face.

The only thing that seems to change this is consequences. Take a random person and just ask them to do something, and whether they do it or not is just based on what they personally want. But when there's a law that tells them to do it, and enforcement of consequences if they don't, suddenly that random person is doing what they're supposed to. A motivation to do the right thing. It's still not a guarantee, but more often than not they'll work to avoid the consequences.

Therefore if you want software projects to stop failing, create laws that enforce doing the things in the project to ensure it succeeds. Create consequences big enough that people will actually do what's necessary. Like a law, that says how to build a thing to ensure it works, and how to test it, and then an independent inspection to ensure it was done right. Do that throughout the process, and impose some kind of consequence if those things aren't done. (the more responsibility, the bigger the consequence, so there's motivation commensurate with impact)

That's how we manage other large-scale physical projects. Of course those aren't guaranteed to work; large-scale public works projects often go over-budget and over-time. But I think those have the same flaw, in that there isn't enough of a consequence for each part of the process to encourage humans to do the right thing.

I guess that’s the real problem I have with SV’s endemic ageism.

I was personally offended, when I encountered it, myself, but that’s long past.

I just find it offensive, that experience is ignored, or even shunned.

I started in hardware, and we all had a reverence for our legacy. It did not prevent us from pursuing new/shiny, but we never ignored the lessons of the past.

right now the industry is spending billions / trillions of $ on to train A.I on badly written open source code.

in universities we teach kids - DSA - but never about to really think about scoping work nor even the unix principle of how software should compose, how to prevent errors etc. hell how many working professionals know about the 10 NASA principles & actually use them in practice ?

we encourage the bright kids to go work at places which are the cathedrals of complexity - but never seeking simple solutions. & again merchants of complexity get paid more - find it easier to find jobs etc.

the tragedy is the failures are documented, but also the fixes to failures as well.

soon enough we're gonna raise a whole generation who doesn't know how to make reliable, robust software from scratch cz of 'vibecoding'. then ultimately civilization collapses.

https://en.wikipedia.org/wiki/Auburn_Dam

https://en.wikipedia.org/wiki/Columbia_River_Crossing

If you're 97% over budget, are you successful? https://en.wikipedia.org/wiki/Big_Dig

FWIW I have read The Phoenix Project and it did help me get a better understanding of "Agile" and the DevOps mindset but since it's not something I apply in my work routinely it's hard to keep it fresh.

My goal is to try and install seeds of success in the small projects I work on and eventually ask questions to get people to think in a similar perspective.

Most sizeable software projects require understanding, in detail, what is needed by the business, what is essential and what is not, and whether any of that is changing over the lifetime of the project. I don't think I've ever been on a project where any of that was known, it was all guess work.

Again those bastards, Fujitsu, were involved. They even sued UK government and won £465 million settlement when their contract was cancelled. But, despite this and their complicity in covering up the failures of the Horizons posts office system, the UK government is still giving them fat contracts.

If senior managers can preside over a massive failure and walk away with a huge pension, there isn't much incentive for them to do better, is there?

Senior stakeholders get into a room, decide they need xyz for the project to really succeed, push this down to managers who in turn try perform miracles with what little development resource they have. Very often they also have an offshore team who is only concerned with prolonging the contract as much as possible, rather than delivering. 2 weeks later senior stakeholders get back into the room...

https://www.csus.edu/indiv/v/velianitis/161/chaosreport.pdf

Also, anything that T. Capers Jones wrote. The most comprehensive one of these books is this:

Estimating Software Costs: Bringing Realism to Estimating Hardcover ISBN-13978-0071483001

Many believe the official recognition of the crisis in developing software were the two NATO conferences in 1968 and 1969.

See the Wikipedia article on the History of Software Engineering.

There have been two small scale experimental comparisons of the waterfall formal model (requirements, design, code, test) and the more prototyping and agile method. They seem to have the same productivity in terms of lines per programmer-month but the formal method tends to produce larger software.

Okay but how much more software is used ? If IT spending has tripled since 2005 but we use 10x more software I'd say the trend is good.

[1] https://www.amazon.com/-/en/dp/B0B63ZG71H

[2] https://www.scribd.com/document/826859800/How-Big-Things-Get...

Then in 2010s they spent $185M on a customized version of IBM's PeopleSoft that was managed directly by a government agency https://en.wikipedia.org/wiki/Phoenix_pay_system

And now in 2020s they are going to spend $385M integrating an existing SaaS made by https://en.wikipedia.org/wiki/Dayforce

That's probably one of the worst and longest software failures in history.

Somehow I come away skeptical of the inevitable conclusion that Phoenix was doomed to fail and instead that perhaps they were hamstrung by architecture constraints dictated by assholes.

Granted, this is an exceedingly hard problem, and I suppose there's some value in reminding ourselves of it - but I'd much rather read thoughts on how to do it better, not just complaints that we're doing it poorly.

You update the system for one small piece, while reconciling with the larger system. Then replace other pieces over time, broadening your scope until you have improved the entire system. There is no other way to succeed without massive pain.

Most users and most management of software projects live in denial that the norm is dystopia.

I can’t help think of any required and useful feature that has happened in computer usage since the early days.

Easier to swallow is that the user interface of desktop operating systems hasn’t changed fundamentally in many years, yet hardware requirements continue to grow.

But even the invention of a mouse requires excessive movement to move a pointer to click on something that a key combination could’ve done much more quickly. The original intention of the mouse was just as another device to use, not necessarily a primary device to direct the majority of workflow.

For software, I am not sure this is helpful. Maybe we would develop way less trash software, if it was different. But then again we would probably still develop engagement farming software, because people would still use or buy that.

I have “magical moments” with these tools, sometimes they solve bugs and implement features in 5 minutes that I couldn’t do in a day… at the same time, quite often they are completely useless and cause you to waste time explaining things that you could probably just code yourself much faster.

So far I believe that there has been too much emphasis in education on coding and algorithms (small scale/tactical stuff) and not enough emphasis on the engineering side of things like version control, QA, system design, management etc. I think the situation has changed (40 years ago most professional programmers didn't even know what version control was, let alone use vc systems) but the scope of the projects has increased faster than our skills and tools.

Lots to break down in this article other than this initial quotation, but I find a lot of parallels in failing software projects, this attitude, and my recent hyper-fixation (seems to spark up again every few years), the sinking of the Titanic.

It was a combination of failures like this. Why was the captain going full speed ahead into a known ice field? Well, the boat can't sink and there (may have been) organizational pressure to arrive at a certain time in new york (aka, imaginary deadline must be met). Why wasn't there enough life jackets and boats for crew and passengers? Well, the boat can't sink anyway, why worry about something that isn't going to happen? Why train crew on how to deploy the life rafts and emergency procedures properly? Same reason. Why didn't the SS Californian rescue the ship? Well, the 3rd party Titanic telegraph operators had immense pressure to send telegrams to NY, and the chatter about the ice field got on their nerves and they mostly ignored it (misaligned priorities). If even a little caution and forward thinking was used, the death toll would have been drastically lower if not nearly nonexistent. It took 2 hours to sink, which is plenty of time to evacuate a boat of that size.

Same with software projects - they often fail over a period of multiple years and if you go back and look at how they went wrong, there often are numerous points and decisions made that could have reversed course, yet, often the opposite happens - management digs in even more. Project timelines are optimistic to the point of delusion and don't build in failure/setbacks into schedules or roadmaps at all. I've had to rescue one of these projects several years ago and it took a toll on me I'm pretty sure I carry to this day, I'm wildly cynical of "project management" as it relates to IT/devops.

So, what's the point here, exactly? "Only licensed engineers as codified by (local!) law are allowed to do projects?" Nah, can't be it, their track record still has too many failures, sometimes even spectacularly explosive and/or implosive ones.

"Any public project should only follow Best Practices"? Sure... "And only make The People feel good"... Incoherent!

Ehhm, so, yeah, maybe things are just complicated, and we should focus more on the amount of effort we're prepared to put in, the competency (c.q. pay grade) of the staff we're willing to assign, and exactly how long we're willing to wait prior to conceding defeat?

?? I don’t think if thinks stopped advancing in terms of significant model improvements that actual utility would be saturated for a while. We have barely begun to consolidate potential into the tooling, use cases, knowledge sharing and depth of that knowledge throughout the workforce on how to make best use.

If someone is looking at AI as a monolith in thing and thinking “oh, silver bullet to the problems of enterprise software etc” then, I really don’t know what to say except that’s on them, not on any true big claims being pushed unless you’re breaking out the long ladders to pick those cherries, or listening to people whose background and placement within things clearly makes them a bad messenger.

Software is not the same as building in the physical world where we get economies of scale.

Building 1,000 bridges will make the cost of the next incremental bridge cheaper due to a zillion factors, even if Bridge #1 is built from sticks (we'll learn standards, stable, fundamental engineering principles, predicable failure modes, etc.) we'll eventually reach a stable, repeatable, scalable approach to build bridges. They will very rarely (in modernity) catastrophically fail (yes, Tacoma Narrows happened but in properly functioning societies it's rare.)

Nobody will say "I want to build a bridge upside-down, out of paper clips and can withstand a 747 driving over it". Because that's physically impossible. But nothing's impossible in software.

Software isn't scalable in this way. It's not scalable because it doesn't have hard constraints (like the laws of physics) - so anything goes and can be in scope; and since writing and integrating large amounts of code is a communication exercise, suffers from diseconomies of scale.

Customers want the software to do exactly what they want and - within reason - no laws of physics are violated if you move a button or implement some business process.

Because everyone wants to keep working the way they want to work, no software project (even if it sounds the same) is the same. Your company's bespoke accounting software will be different than mine, even if we are direct competitors in the same market. Our business processes are different, org structures are different, sales processes are different, etc.. So they all build different accounting software, even if the fundamentals (GaaP, double-entry bookkeeping, etc.) are shared.

It's also the same reason why enterprise software sucks - do you think that a startup building expense management starts off being a giant mess of garbage? No! IT starts off simple and clean and beautiful because their initial customer base (startups) are beggars and cannot be choosers, so they adapt their process to the tool. But then larger companies come along with dissimilar requirements and, Expense Management SaaS Co. wins that deal by changing the product to work with whatever oddball requirements they have, and so on, until the product essentially is a bunch of config options and workflows that you have to build yourself.

(Interestingly, I think these products become asymptotically stuck - any feature you add or remove will make some of your customers happy and some of your customers mad, so the product can never get "better" globally).

We can have all the retrospectives and learnings we want but the goal - "Build big software" - is intractable, and as long as we keep trying to do that, we will inevitably fail. This is not a systems problem that we can fix.

The lesson is: "never build big software".

(Small software is stuff like Bezos' two pizza team w/APIs etc. - many small things make a big thing)

Most advertising campaigns fail.

Most failed projects I've been around were killed by accidental complexity—too many abstraction layers wrapping simple logic.

I trust my phone to work so much that it is now the single, non-redundant source for keys to my apartment, keys to my car, and payment method. Phones could only even hope to do all of these things as of like ~4 years ago, and only as of ~this year do I feel confident enough to not even carry redundancies. My phone has never breached that trust so critically that I feel I need to.

Of course, this article talks about new software projects. And I think the truth and reason of the matter lies in this asymmetry: Android/iOS are not new. Giving an engineering team agency and a well-defined mandate that spans a long period of time oftentimes produces fantastic software. If that mandate often changes; or if it is unclear in the first place; or if there are middlemen stakeholders involved; you run the risk of things turning sideways. The failure of large software systems is, rarely, an engineering problem.

But, of course, it sometimes is. It took us ~30-40 years of abstraction/foundation building to get to the pretty darn good software we have today. It'll take another 30-40 years to add one or two more nines of reliability. And that's ok; I think we're trending in the right direction, and we're learning. Unless we start getting AI involved; then it might take 50-60 years :)

> “A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.”

[0]: https://personalmba.com/galls-law/

https://queue.acm.org/detail.cfm?id=3489045

https://therecord.media/cybersecurity-software-liability-sta...

Computer Weekly first broke the story, for which they deserve much credit. But I believe Private Eye did much of the long term campaigning.

So we added a language and cultural barrier, 12 hour offset, and thousands of miles of separation with outsourcing.

Software was failing and mismanaged.

So now we will take the above failures, and now tack on an AI "prompt engineering" barrier (done by the above outsourced labor).

And on top of that, all engineers that know what they are doing are devalued from the market, all the newer engineers will be AI braindead.

Everything will be fixed!

there you are why its failling, the fact that these system is overly massive and complex that sometimes the original creator and architecture that design this system cant foresee the future needs and etc

You could says its incompetence but the fact that software change so much is last 20 years make it most people cant really design a "future proof" system in a way that it cant cause trouble in the future

In the same way that hardware improvements are quickly gobbled up by more demanding software.

The people doing the programming will also be more removed technically. I can do Python, Java , Kotlin. I can do a little C++ ,less C, and a lot less assembly.

The second problem are big con.

First, we as a society should really be scrutinizing what we invest in. Trillions of dollars could end homelessness as a rounding error.

Second, real people are going to be punished for this as the layoffs go into overdrive, people lose their houses and people struggle to have enough to eat.

Third, the ultimate goal of all this investment is to displace people from the labor pool. People are annoying. They demand things like fair pay, safe working conditions and sick leave.

Who will buy the results of all this AI if there’s no one left with a job?

Lastly, the externalities of all this investment are indefensible. For example, air and water pollution and rising utility prices.

We’re bouldering towards a future with a few thousand wealthy people where everyone else lives in worker housing, owns nothing and is the next incarnation of brick kiln workers on wealthy estates.

These projects are often characterized by very complex functional requirements, yet are undertaken by those who primarily only know (and endlessly argue about) non-functional requirements.

There should be things like professional certifications that engineers have to maintain through continuous education, a professional code of ethics, a board of review, and other functions.

My reasoning is that we are at the point where a software "engineer" can make a mistake that can have the same impact as a civil engineer making a bad calculation and causing a bridge collapse.

There's different levels to this, of course. An app for booking restaurant reservations wouldn't need that much oversight. But we've seen some outages having massive impacts that quite frankly did not happy twenty years ago.

Nuclear reactor projects seem to be regularly delivered massively late and over budget.

But sure, blame the community at large, not the execs who make bad decisions in the name of short-term profits, then fail upward with golden parachutes into their next gig.

And definitely don't blame government for not punishing egregious behavior of corporations. Don't worry, you get a year of free credit monitoring from some scummy company who's also selling your data. Oh and justice was served to the offending corp, who got a one-time fine of $300k, when they make billions every quarter.

Maybe if we just outsource everything to AI, consultants, and offshore sweat shops things will improve!

Okay cool, good article.

There are no generic, simple solutions for complex IT challenges. But there are ground rules for finding and implementing simple solutions. I have created a playbook to prevent IT diasasters, The art and science towards simpler IT solutions see https://nocomplexity.com/documents/reports/SimplifyIT.pdf

translation: "leave it to us professionals". Gate-keeping of this kind is exactly how computer science (the one remaining technical discipline still making reliable progress) could become like all of the other anemic, cursed fields of engineering. people thinking "hey im pretty sure I could make a better version of this" and then actually doing it is exactly how progress happens. I hope nobody reads this article and takes it seriously

absent understanding, large companies engage in cargo cult behaviors: they create a sensible org chart, produce a gannt chart, have the coders start whacking code, presumably in 9 months a baby comes out.

every time, ugly baby

1. Connecting pay to work - estimates (replanning is learning, not failure)

2. Connecting work to pay - management (the world is fractal-like, scar tissue and band-aids)

I do not pre-suppose that there are definite solutions to these problems - there may be solutions, but getting there may require going far out of our way. As the old farmer said "Oh, I can tell you how to get there, but if I was you, I wouldn't start from here"

1. Pay to Work - someone is paying for the software project, and they need to know how much it will cost. Thus estimates are asked for, an architecture is asked for, and the architecture is tied to the estimates.

This is 'The Plan!'. The project administrators will pick some lifecycle paradigm to tie the architecture to the cost estimate.

The implementation team will learn as they do their work. This learning is often viewed as failure, as the team will try things that don't work.

The implementation team will learn that the architecture needs to change in some large ways and many small ways. The smallest changes are absorbed in regular work. Medium and Large changes will require more time (thus money); This request for more money will be viewed as a failure in estimation and not as learning.

2. Work to Pay - as the architecture is implemented, development tasks are completed. The Money People want Numbers, because Money People understand how they feel about Numbers. Also these Numbers will talk to other Numbers outside the company. Important Numbers with names like Share Price.

Thus many layers of management are chartered and instituted. The lowest layer of management is the self-managed software developer. The software developer will complete development tasks related to the architecture, tied to the plan, attached to the money (and the spreadsheets grew all around, all around [0]).

When the developer communicates about work, the Management Chain cares to hear about Numbers, but sometimes they must also involve themselves in failures.

It is bad to fail, especially repeated failures at the same kind of task. So managers institute rules to prevent failures. These rules are put in a virtual cabinet, or bureau. Thus we have Rules of the Bureau or Bureaucracy. These rules are not morally bad or good; not factually incorrect or correct, but whenever we notice them, they feel bad; We notice the ones that feel bad TO US. We are often in favor of rules that feel bad to SOMEONE ELSE. You are free to opt out of this system, but there is a price to doing so.

----

Too much writing, I desist from decoding verbiage:

Thus it is OK for individuals to learn many small things, but it is a failure for the organization to learn large things. Trying to avoid and prevent failure is viewed as admirable; trying to avoid learning is self-defeating.

----

0. https://www.google.com/search?q=the+green+grass+grew+all+aro...

> git commit -am "decomposing recapitulating and recontextualizing software development bureaucracy" && git push

Take "AI", for instance. It is being adopted left and right as if it's the solution to all of our problems, and developers, managers, and executives are increasingly relying on it. Companies and governments love it because it can cut costs and potentially make us more productive. Most people are more than happy to offload their work to it, do a cursory check of its output, if at all, and ship it or publish it and claim the work as their own. After all, it can always serve as a scapegoat if things do go wrong, and its manufacturers can brush these off as user errors. Ultimately there is no accountability.

These are all components of a recipe for greater disasters. As these tools are adopted in industries where safety is paramount, in the military, etc., it's only a matter of time for more human lives to be impacted. Especially now when more egomaniacal autocrats are taking power, and surrounding themselves with yes-people. Losing face and admitting failure is not part of their playbook. We're digging ourselves into a hole we might not be able to get out of.

1. Enable grift to cronies

2. Promo-driven culture

3. Resume-oriented software architecture

* Construction and Engineering -- Massive cost overruns and schedule delays on large infrastructure projects (e.g., public transit systems, bridges)

* Military and Government -- Defense acquisition programs notorious for massive cost increases and years-long delays, where complex requirements and bureaucratic processes create an environment ripe for failure.

* Healthcare -- Hospital system implementations or large research projects that exceed budgets and fail to deliver intended efficiencies, often due to resistance to change and poor executive oversight.

It's leadership and accountability (well, the lack of them).

https://youtu.be/D43PlUr1x_E?si=em2nNYuI8WDvtP21

Software is also incredibly hard, the human mind can understand the physical space very well but once we're deep into abstractions it simply struggles to keep up with it.

It is easier to explain how to build a house from scratch to virtually anyone than a mobile app/Excel.

> Phoenix project executives believed they could deliver a modernized payment system, customizing PeopleSoft’s off-the-shelf payroll package to follow 80,000 pay rules spanning 105 collective agreements with federal public-service unions. It also was attempting to implement 34 human-resource system interfaces across 101 government agencies and departments required for sharing employee data.

So basically people -- none of them in IT, but rather working for the government -- built something extraordinarily complex (80k rules!), and then are like wow, it's unforeseen that would make anything downstream at least equally as complex. And then the article blames IT in general. When this data point tells us that replacing a business process that used to require (per [1]) 2,000 pay advisors to perform will be complex. While working in an organization that has shit the bed so thoroughly that paying its employees requires 2k people. For an organization of 290k, so 0.6% of headcount is spent on paying employees!

IT is complex, but incompetent people and incompetent orgs do not magically become competent when undertaking IT projects.

Also too, making extraordinarily complex things they shouting the word "computer" at them like you're playing D&D and it's a spell does not make them simple.

[1] https://www.oag-bvg.gc.ca/internet/English/parl_oag_201711_0...

If companies had hired real developers instead of cosplayers who are stunlocked with imposter syndrome as the only candidate pool with time to memorize a rainbow table of arbitrary game trivia questions and answers, things would actually work.

* Formatting

* Style

* Conventions

* Patterns

* Using the latest frameworks or whats en-vogue

I think where I've seen results delivered effectively and consistently is where there is a universal style enforced, which removes the individualism from the codebase. Some devs will not thrive in that environment, but instead it makes the code a means-to-the-end, rather than being-the-end.