Need is a strong word. That's why I said effective.

You can often iterate to something that kind of works by adding epicycles. What you're left with is something that fails in rare cases you can't explain, is difficult to change (we don't touch that code), and is slow.

Compare a complex homegrown data store to a relational database like postgresSQL. Both get the job done, but one has significantly more conceptual clarity and reliability.

Being able to come across a hard problem and say, ok this is how to frame it and here is how to go about it, turns months of fiddling into a direct route.

> Compare a complex homegrown data store to a relational database like postgresSQL. Both get the job done, but one has significantly more conceptual clarity and reliability.

That's a good example of what I said. Most software engineers don't develop new database management systems. They just use one. And if they merely use it, they don't need or benefit from nontrivial math. The math is abstracted away behind the intuitive SQL syntax.

So did postgres appear out of thin air? What new SW technologies need to be developed?

If you're not interested in acquiring the kind of competency to develop new software, and are just interested in combining existing software, then you don't need a college degree.