HN Reader

It's also hard when the team actually cares, but there are skills you can learn. Early in my career, I got into solving some of the barriers to software project management (e.g., requirements analysis and otherwise understanding needs, sustainable architecture, work breakdown, estimation, general coordination, implementation technology).

But once you're a bit comfortable with the art and science of those, big new challenges are more about political and environment reality. It comes down to alignment and competence of: workers, internal team leadership, partners/vendors, customers, and investors/execs.

Discussing this is a little awkward, but maybe start with alignment, since most of the competence challenges are rooted in mis-alignments: never developing nor selecting for the skills that alignment would require.

I guess that’s the real problem I have with SV’s endemic ageism.

I was personally offended, when I encountered it, myself, but that’s long past.

I just find it offensive, that experience is ignored, or even shunned.

I started in hardware, and we all had a reverence for our legacy. It did not prevent us from pursuing new/shiny, but we never ignored the lessons of the past.

The only thing that seems to change this is consequences. Take a random person and just ask them to do something, and whether they do it or not is just based on what they personally want. But when there's a law that tells them to do it, and enforcement of consequences if they don't, suddenly that random person is doing what they're supposed to. A motivation to do the right thing. It's still not a guarantee, but more often than not they'll work to avoid the consequences.

Therefore if you want software projects to stop failing, create laws that enforce doing the things in the project to ensure it succeeds. Create consequences big enough that people will actually do what's necessary. Like a law, that says how to build a thing to ensure it works, and how to test it, and then an independent inspection to ensure it was done right. Do that throughout the process, and impose some kind of consequence if those things aren't done. (the more responsibility, the bigger the consequence, so there's motivation commensurate with impact)

That's how we manage other large-scale physical projects. Of course those aren't guaranteed to work; large-scale public works projects often go over-budget and over-time. But I think those have the same flaw, in that there isn't enough of a consequence for each part of the process to encourage humans to do the right thing.

FWIW I have read The Phoenix Project and it did help me get a better understanding of "Agile" and the DevOps mindset but since it's not something I apply in my work routinely it's hard to keep it fresh.

My goal is to try and install seeds of success in the small projects I work on and eventually ask questions to get people to think in a similar perspective.

https://en.wikipedia.org/wiki/Auburn_Dam

https://en.wikipedia.org/wiki/Columbia_River_Crossing

If you're 97% over budget, are you successful? https://en.wikipedia.org/wiki/Big_Dig

Most sizeable software projects require understanding, in detail, what is needed by the business, what is essential and what is not, and whether any of that is changing over the lifetime of the project. I don't think I've ever been on a project where any of that was known, it was all guess work.

But sure, blame the community at large, not the execs who make bad decisions in the name of short-term profits, then fail upward with golden parachutes into their next gig.

And definitely don't blame government for not punishing egregious behavior of corporations. Don't worry, you get a year of free credit monitoring from some scummy company who's also selling your data. Oh and justice was served to the offending corp, who got a one-time fine of $300k, when they make billions every quarter.

Maybe if we just outsource everything to AI, consultants, and offshore sweat shops things will improve!

Okay cool, good article.

* Construction and Engineering -- Massive cost overruns and schedule delays on large infrastructure projects (e.g., public transit systems, bridges)

* Military and Government -- Defense acquisition programs notorious for massive cost increases and years-long delays, where complex requirements and bureaucratic processes create an environment ripe for failure.

* Healthcare -- Hospital system implementations or large research projects that exceed budgets and fail to deliver intended efficiencies, often due to resistance to change and poor executive oversight.

[1] https://www.amazon.com/-/en/dp/B0B63ZG71H

[2] https://www.scribd.com/document/826859800/How-Big-Things-Get...

Then in 2010s they spent $185M on a customized version of IBM's PeopleSoft that was managed directly by a government agency https://en.wikipedia.org/wiki/Phoenix_pay_system

And now in 2020s they are going to spend $385M integrating an existing SaaS made by https://en.wikipedia.org/wiki/Dayforce

That's probably one of the worst and longest software failures in history.

I have “magical moments” with these tools, sometimes they solve bugs and implement features in 5 minutes that I couldn’t do in a day… at the same time, quite often they are completely useless and cause you to waste time explaining things that you could probably just code yourself much faster.

Granted, this is an exceedingly hard problem, and I suppose there's some value in reminding ourselves of it - but I'd much rather read thoughts on how to do it better, not just complaints that we're doing it poorly.

So, what's the point here, exactly? "Only licensed engineers as codified by (local!) law are allowed to do projects?" Nah, can't be it, their track record still has too many failures, sometimes even spectacularly explosive and/or implosive ones.

"Any public project should only follow Best Practices"? Sure... "And only make The People feel good"... Incoherent!

Ehhm, so, yeah, maybe things are just complicated, and we should focus more on the amount of effort we're prepared to put in, the competency (c.q. pay grade) of the staff we're willing to assign, and exactly how long we're willing to wait prior to conceding defeat?

The second problem are big con.

Software is not the same as building in the physical world where we get economies of scale.

Building 1,000 bridges will make the cost of the next incremental bridge cheaper due to a zillion factors, even if Bridge #1 is built from sticks (we'll learn standards, stable, fundamental engineering principles, predicable failure modes, etc.) we'll eventually reach a stable, repeatable, scalable approach to build bridges. They will very rarely (in modernity) catastrophically fail (yes, Tacoma Narrows happened but in properly functioning societies it's rare.)

Nobody will say "I want to build a bridge upside-down, out of paper clips and can withstand a 747 driving over it". Because that's physically impossible. But nothing's impossible in software.

Software isn't scalable in this way. It's not scalable because it doesn't have hard constraints (like the laws of physics) - so anything goes and can be in scope; and since writing and integrating large amounts of code is a communication exercise, suffers from diseconomies of scale.

Customers want the software to do exactly what they want and - within reason - no laws of physics are violated if you move a button or implement some business process.

Because everyone wants to keep working the way they want to work, no software project (even if it sounds the same) is the same. Your company's bespoke accounting software will be different than mine, even if we are direct competitors in the same market. Our business processes are different, org structures are different, sales processes are different, etc.. So they all build different accounting software, even if the fundamentals (GaaP, double-entry bookkeeping, etc.) are shared.

It's also the same reason why enterprise software sucks - do you think that a startup building expense management starts off being a giant mess of garbage? No! IT starts off simple and clean and beautiful because their initial customer base (startups) are beggars and cannot be choosers, so they adapt their process to the tool. But then larger companies come along with dissimilar requirements and, Expense Management SaaS Co. wins that deal by changing the product to work with whatever oddball requirements they have, and so on, until the product essentially is a bunch of config options and workflows that you have to build yourself.

(Interestingly, I think these products become asymptotically stuck - any feature you add or remove will make some of your customers happy and some of your customers mad, so the product can never get "better" globally).

We can have all the retrospectives and learnings we want but the goal - "Build big software" - is intractable, and as long as we keep trying to do that, we will inevitably fail. This is not a systems problem that we can fix.

The lesson is: "never build big software".

(Small software is stuff like Bezos' two pizza team w/APIs etc. - many small things make a big thing)

Lots to break down in this article other than this initial quotation, but I find a lot of parallels in failing software projects, this attitude, and my recent hyper-fixation (seems to spark up again every few years), the sinking of the Titanic.

It was a combination of failures like this. Why was the captain going full speed ahead into a known ice field? Well, the boat can't sink and there (may have been) organizational pressure to arrive at a certain time in new york (aka, imaginary deadline must be met). Why wasn't there enough life jackets and boats for crew and passengers? Well, the boat can't sink anyway, why worry about something that isn't going to happen? Why train crew on how to deploy the life rafts and emergency procedures properly? Same reason. Why didn't the SS Californian rescue the ship? Well, the 3rd party Titanic telegraph operators had immense pressure to send telegrams to NY, and the chatter about the ice field got on their nerves and they mostly ignored it (misaligned priorities). If even a little caution and forward thinking was used, the death toll would have been drastically lower if not nearly nonexistent. It took 2 hours to sink, which is plenty of time to evacuate a boat of that size.

Same with software projects - they often fail over a period of multiple years and if you go back and look at how they went wrong, there often are numerous points and decisions made that could have reversed course, yet, often the opposite happens - management digs in even more. Project timelines are optimistic to the point of delusion and don't build in failure/setbacks into schedules or roadmaps at all. I've had to rescue one of these projects several years ago and it took a toll on me I'm pretty sure I carry to this day, I'm wildly cynical of "project management" as it relates to IT/devops.

Most advertising campaigns fail.

absent understanding, large companies engage in cargo cult behaviors: they create a sensible org chart, produce a gannt chart, have the coders start whacking code, presumably in 9 months a baby comes out.

every time, ugly baby

You update the system for one small piece, while reconciling with the larger system. Then replace other pieces over time, broadening your scope until you have improved the entire system. There is no other way to succeed without massive pain.

There are no generic, simple solutions for complex IT challenges. But there are ground rules for finding and implementing simple solutions. I have created a playbook to prevent IT diasasters, The art and science towards simpler IT solutions see https://nocomplexity.com/documents/reports/SimplifyIT.pdf

translation: "leave it to us professionals". Gate-keeping of this kind is exactly how computer science (the one remaining technical discipline still making reliable progress) could become like all of the other anemic, cursed fields of engineering. people thinking "hey im pretty sure I could make a better version of this" and then actually doing it is exactly how progress happens. I hope nobody reads this article and takes it seriously

I trust my phone to work so much that it is now the single, non-redundant source for keys to my apartment, keys to my car, and payment method. Phones could only even hope to do all of these things as of like ~4 years ago, and only as of ~this year do I feel confident enough to not even carry redundancies. My phone has never breached that trust so critically that I feel I need to.

Of course, this article talks about new software projects. And I think the truth and reason of the matter lies in this asymmetry: Android/iOS are not new. Giving an engineering team agency and a well-defined mandate that spans a long period of time oftentimes produces fantastic software. If that mandate often changes; or if it is unclear in the first place; or if there are middlemen stakeholders involved; you run the risk of things turning sideways. The failure of large software systems is, rarely, an engineering problem.

But, of course, it sometimes is. It took us ~30-40 years of abstraction/foundation building to get to the pretty darn good software we have today. It'll take another 30-40 years to add one or two more nines of reliability. And that's ok; I think we're trending in the right direction, and we're learning. Unless we start getting AI involved; then it might take 50-60 years :)

Somehow I come away skeptical of the inevitable conclusion that Phoenix was doomed to fail and instead that perhaps they were hamstrung by architecture constraints dictated by assholes.

1. Connecting pay to work - estimates (replanning is learning, not failure)

2. Connecting work to pay - management (the world is fractal-like, scar tissue and band-aids)

I do not pre-suppose that there are definite solutions to these problems - there may be solutions, but getting there may require going far out of our way. As the old farmer said "Oh, I can tell you how to get there, but if I was you, I wouldn't start from here"

1. Pay to Work - someone is paying for the software project, and they need to know how much it will cost. Thus estimates are asked for, an architecture is asked for, and the architecture is tied to the estimates.

This is 'The Plan!'. The project administrators will pick some lifecycle paradigm to tie the architecture to the cost estimate.

The implementation team will learn as they do their work. This learning is often viewed as failure, as the team will try things that don't work.

The implementation team will learn that the architecture needs to change in some large ways and many small ways. The smallest changes are absorbed in regular work. Medium and Large changes will require more time (thus money); This request for more money will be viewed as a failure in estimation and not as learning.

2. Work to Pay - as the architecture is implemented, development tasks are completed. The Money People want Numbers, because Money People understand how they feel about Numbers. Also these Numbers will talk to other Numbers outside the company. Important Numbers with names like Share Price.

Thus many layers of management are chartered and instituted. The lowest layer of management is the self-managed software developer. The software developer will complete development tasks related to the architecture, tied to the plan, attached to the money (and the spreadsheets grew all around, all around [0]).

When the developer communicates about work, the Management Chain cares to hear about Numbers, but sometimes they must also involve themselves in failures.

It is bad to fail, especially repeated failures at the same kind of task. So managers institute rules to prevent failures. These rules are put in a virtual cabinet, or bureau. Thus we have Rules of the Bureau or Bureaucracy. These rules are not morally bad or good; not factually incorrect or correct, but whenever we notice them, they feel bad; We notice the ones that feel bad TO US. We are often in favor of rules that feel bad to SOMEONE ELSE. You are free to opt out of this system, but there is a price to doing so.

----

Too much writing, I desist from decoding verbiage:

Thus it is OK for individuals to learn many small things, but it is a failure for the organization to learn large things. Trying to avoid and prevent failure is viewed as admirable; trying to avoid learning is self-defeating.

----

0. https://www.google.com/search?q=the+green+grass+grew+all+aro...

> git commit -am "decomposing recapitulating and recontextualizing software development bureaucracy" && git push

1. Enable grift to cronies

2. Promo-driven culture

3. Resume-oriented software architecture

Software is also incredibly hard, the human mind can understand the physical space very well but once we're deep into abstractions it simply struggles to keep up with it.

It is easier to explain how to build a house from scratch to virtually anyone than a mobile app/Excel.

https://youtu.be/D43PlUr1x_E?si=em2nNYuI8WDvtP21

It's leadership and accountability (well, the lack of them).

In the same way that hardware improvements are quickly gobbled up by more demanding software.

The people doing the programming will also be more removed technically. I can do Python, Java , Kotlin. I can do a little C++ ,less C, and a lot less assembly.

So we added a language and cultural barrier, 12 hour offset, and thousands of miles of separation with outsourcing.

Software was failing and mismanaged.

So now we will take the above failures, and now tack on an AI "prompt engineering" barrier (done by the above outsourced labor).

And on top of that, all engineers that know what they are doing are devalued from the market, all the newer engineers will be AI braindead.

Everything will be fixed!

First, we as a society should really be scrutinizing what we invest in. Trillions of dollars could end homelessness as a rounding error.

Second, real people are going to be punished for this as the layoffs go into overdrive, people lose their houses and people struggle to have enough to eat.

Third, the ultimate goal of all this investment is to displace people from the labor pool. People are annoying. They demand things like fair pay, safe working conditions and sick leave.

Who will buy the results of all this AI if there’s no one left with a job?

Lastly, the externalities of all this investment are indefensible. For example, air and water pollution and rising utility prices.

We’re bouldering towards a future with a few thousand wealthy people where everyone else lives in worker housing, owns nothing and is the next incarnation of brick kiln workers on wealthy estates.

> Phoenix project executives believed they could deliver a modernized payment system, customizing PeopleSoft’s off-the-shelf payroll package to follow 80,000 pay rules spanning 105 collective agreements with federal public-service unions. It also was attempting to implement 34 human-resource system interfaces across 101 government agencies and departments required for sharing employee data.

So basically people -- none of them in IT, but rather working for the government -- built something extraordinarily complex (80k rules!), and then are like wow, it's unforeseen that would make anything downstream at least equally as complex. And then the article blames IT in general. When this data point tells us that replacing a business process that used to require (per [1]) 2,000 pay advisors to perform will be complex. While working in an organization that has shit the bed so thoroughly that paying its employees requires 2k people. For an organization of 290k, so 0.6% of headcount is spent on paying employees!

IT is complex, but incompetent people and incompetent orgs do not magically become competent when undertaking IT projects.

Also too, making extraordinarily complex things they shouting the word "computer" at them like you're playing D&D and it's a spell does not make them simple.

[1] https://www.oag-bvg.gc.ca/internet/English/parl_oag_201711_0...

* Formatting

* Style

* Conventions

* Patterns

* Using the latest frameworks or whats en-vogue

I think where I've seen results delivered effectively and consistently is where there is a universal style enforced, which removes the individualism from the codebase. Some devs will not thrive in that environment, but instead it makes the code a means-to-the-end, rather than being-the-end.