What’s To Be Done About The Reproducibility Crisis In The Social Sciences?

There has of late been a lot of discussion about the future of social science and the direction that it should take if it is to remain relevant. This has been touched off by a “reproducibility crisis” in social science which has brought its system of validating results into serious question. Perhaps the most noteworthy headline generated by the matter involved an investigation, published in the journal Science, which found that over 60% of a selection of peer-reviewed research papers in psychology that had been published in prestigious journals in the field had claimed results that were not reproducible. Similar bad news emanates from other areas of the social sciences, throwing doubt on the reliability and usefulness of the entire field.

But all of this only confirms what traditionalists and reactionaries already knew: that a very large amount of the social science that has been produced over the past century – or even longer – is utter bunk. News of the reproducibility crisis provides some nice validation, but it is hardly a surprise to us that the social sciences are fundamentally broken. The question is what to do about it. In an article in the Drexel University-affiliated online magazine The Smart Set, the economist Michael Lind recently proposed abolishing social science entirely*.

Lind’s complaints are reasonably valid: That social scientists have applied the mindset and methodology of physics in areas of study to which they are completely unsuited; that once social science became convinced that it was a “hard science” it became arrogant; that this has caused a disconnect from reality and a prioritization of abstract theory over practical results. Yet, abandoning social science altogether seems unwise. For one thing, good social science is needed in order to combat bad social science, which despite the reproducibility crisis retains tremendous influence. For another, with the power of faith and tradition broken or nearly so throughout the West, something is needed in order to guide us forward (even if its goal is to eventually lead us back to faith and tradition). Lastly, a full documenting of what went wrong with the West and how it happened is necessary so that we can (hopefully) avoid making quite exactly the same mistakes again in the future.

So no, I don’t propose a complete abandonment of the field of social science. What I do propose is a complete reorientation of it around a new model, or at very least (as we cannot control what leftists do), the adoption of a new model by social scientists of the right. In short, while the left’s social science takes physics as its model, the right’s social science should take engineering as its model.

One may be tempted to ask: What’s the difference? After all, engineering is no more than applied physics, isn’t it? Well, yes it is, but the qualifier “applied” makes all the difference in the world. It imbues engineering with an entirely different mindset–one far more suited to the task of building functioning things in the real world. While physics is theoretical, engineering is concrete. While physics explores what is possible, engineering explores what is practical. While physics asks: “Is it theoretically plausible that X could work under exactly perfect conditions?”; engineering asks: “Can we design X such that it will stand up to real-world conditions?”**. While physicists theorize, engineers test.

This should sound like a godsend to the men of the West, who have spent 250 years watching their civilization be slowly destroyed in the name of one or another fine-sounding theory – all of them untested, or unfalsifiable, or both – developed by men of impressive credentials.

The misconceptions of the “I Fucking Love Science” crowd aside, the truth is that engineers prefer the known to the unknown, the tested to the untested, the proven to the unproven. They understand that innovation for innovation’s sake is bad engineering. Earthy as it may be, the old saws “If it ain’t broke, don’t fix it”  and “The perfect is the enemy of the good” both reflect sound principles of engineering. If an existing solution is working fine, a good engineer will see no reason to replace it. The fact that the concept behind a new solution sounds fantastic does not move the engineer, because he understands that any slick-talking snake-oil salesman can make big promises – what matters is not what someone can promise, but what they can actually deliver. Yes, sometimes a new design that may work better or that may do something that the old design couldn’t do at all comes along; but when it does, the engineer will insist that it be exhaustively tested before it is put into use.

I have seen this personally. In an earlier part of my life, I was tangentially involved with the testing regime that the FAA put GPS through before they allowed it to be used for instrument approaches in civilian aircraft. By that time, GPS was a well-developed technology that had been successfully used by the military for years. Despite this, the FAA insisted on doing years more worth of testing, which included thousands of hours of field testing in actual airplanes during flight. As far as they were concerned, nobody was going to end up flying a 747 full of vacationers into a mountainside because the FAA hadn’t tested this technology thoroughly enough. This is the engineering mindset at work.

That the left and the IFLS crowd (Or do I repeat myself?) seem not to understand this is a function of the fact that what little they know about engineering they take from the example of Silicon Valley. But the Valley is an exceptionally poor model of engineering-thinking. Silicon Valley ships undertested, half-broken products all the time, because it is in a business that is essentially low-stakes. If the latest release of iOS or Google Hangouts or the Facebook app is a buggy mess, nobody dies because of it. But this is not the case for other technologies. In short, the Valley is a consumer-focused and often ego-driven industry where flashy “innovation” is prioritized, safety concerns are minimal to nonexistent, and there are no stakes for failure other than a little pride and money. Thus, no matter how trendy and high-status it may currently be, it should not be taken as a model for what serious engineering looks like.

In serious engineering, nothing is deployed until it has been tested and tested and tested again, under every possible condition that can be conceived of. Especially, every possible failure mode of a new technology will be explored in depth, both to determine its likelihood and to determine its possible severity. Here it is worth noting of the left that not only do they put forth untested and unfalsifiable theories, but so convinced are they that history is on their side, that their views are backed by Science!, and that their utopia is inevitable, that as a matter of principle they never consider the possibility of failure. This extends to all of their thinking.

One may recall that the reason why Chernobyl (the official name of which was the V. I. Lenin Atomic Energy Station) was such an awful disaster was that the Soviets didn’t believe in constructing containment buildings around their nuclear reactors, because (of course!) Marxist science was so perfect that no such measure would ever be needed. Why think of containing that which cannot fail? This is a perfect metaphor for Marxism itself: let us not forget that Marx claimed that what he had was not an economic theory or a philosophy: no, what he had was hard science – physics itself! – the triumph of which was absolutely inevitable. If modern leftism has budged one inch from this belief about its own theories, I have seen no evidence of it.

(An aside: For a time in the early 00s, I worked in the town of Lawrence, Massachusetts – one of the many old mill towns in the northeast that has fallen on hard times due to a combination of automation and foreign competition. In the early 90s, a team of divers exploring the Merrimack River where it runs through Lawrence discovered a set of water turbines that was built in the 1850s in order to power the mills that sat along the river. It was found that the turbines were still in fine shape, and would only need the mud dredged out of them in order to be put back into service. After that, all the city of Lawrence would need to do would be to buy some generators, put them in one of the long-abandoned mill buildings, hook the generators up to the water turbines, and they would have enough electricity to power the entire city with enough left over that they could sell it at a profit to the power grid. It would be free energy – both clean and renewable for as long as the Merrimack keeps flowing. It is a testament to the incompetence of the Lawrence City Council that nothing ever came of this, but the point stands – sometimes old technology really is better. Certainly, this would have been a better solution for generating electricity than Chernobyl proved to be.)

As with the V. I. Lenin Atomic Energy Station, so with V. I. Lenin’s own history – with both, we see an unproven technology (in one case a scientific technology, in the other a social technology) put into practice by people who insisted that their theoretical model was so perfect that failure was impossible, and thus that both a sufficient amount of testing and the consideration of possible failure modes were foolish and unnecessary. Among those who still defend Marxism (or who advocate the various attempts to rebadge it under another name) we continue to see (a misapplication of) physics thinking. In physics, it is valid to say that the failure of X under conditions Y is no proof whatsoever that it would fail under conditions Z. Every possible set of combinations must be tried until it can be categorically stated on a theoretical level that X does not ever work under any conditions. Engineering, on the other hand, learns from history; it can say: “X failed every time we tried it in the real world; let’s try something different instead”.

And this brings us to a key concept: history. The left, being utopian, theory-based, and incapable of conceiving of the possibility that its ideas might ever fail, is by its nature fundamentally incapable of learning from history. But to the right, history is the answer to the critical question: How is it possible to test social technology? History is the testing ground of social technologies, and tradition represents the lessons that have come of that testing. Over thousands of years, our ancestors – who the left regard as idiots because they did not have iPads, but who we know were generally wiser and more realistic when it comes to human nature than we are – developed, adjusted, and refined the social technologies that they passed down to us in the form of tradition.

Sometimes the lessons that they learned came in the form of the words of wise men; often they came in the form of iron, blood, and fire. But either way, they learned from their experiences, and we of the right learn from them as well, because we understand that while the left never tires of hectoring their adversaries by telling them that science is objectively true (or at least, it is when its results are actually reproducible), it isn’t the only thing that is objectively true – so is history. The left may try to tell us that history is irrelevant or meaningless, and that their untested, unfalsifiable theories*** are superior to our ancestors’ lived experiences and the traditions born of the lessons they learned from them, but we know better. We know that history actually happened, that our ancestors were wise, and that tradition represents tried, tested, proven social technology.

Does this mean that we will never deploy a new social technology, or adjust an existing one, when circumstances demand it? No. But we will not innovate for innovation’s sake; we will not change just to change. Where an existing solution has been shown to work, we will keep using it (or, if one has been abandoned in favor of something that doesn’t work, we will bring it back). Where an existing solution no longer works as well as it once did (and the claim that it doesn’t must be proven, not just asserted), we will adjust it as minimally as possible in order to meet the changed circumstances.

In no case will we throw out a tried, tested, proven social technology in favor of an untried, untested, unproven theory, no matter how good it looks on paper. We will not scrap what works, even if imperfectly so, in exchange for pie-in-the-sky promises of perpetual motion machines that will deliver a free lunch to everyone. We will say that if a theorist wants us to believe their claims of having designed a better system, then they must prove that it is so in the real world. And until they have proven their theories, we will say: do not bother us with them – untested theory is worthless to us****.

With engineering as our model, we will adopt no social technology until it is thoroughly tested, adjust no functioning social technology until it is absolutely necessary to do so, and go back to tried-and-true technologies when new ones fail. We will dredge out old water turbines instead of building new Chernobyls, and appeals to the fact that it is the current year and water turbines are horribly old-fashioned will be meaningless to us. We will go with what we know works, because that’s just good engineering.

(*It appears that universities in Japan are beginning to do just that.)

(**The Constitution is a good example of a document that follows the physics model. It is theoretically possible for the Constitution to govern a certain kind people well, and in fact it did, for a limited period of time. But it was only engineered to stand up to what is essentially a best-case scenario. As soon as those design limits were exceeded, which happened in less than a century, it failed.)

(***It is worth noting that very often the left’s theories have been tested and shown to be failures, but where this is true, they simply raise the standard of proof of failure until it is unreachable, making their theories effectively unfalsifiable. For this, see the claim that the failures of communism in the 20th century are meaningless because those weren’t “real communism”. Where this fails, they will simply lie outright, as in the claim that the atheist left never oppressed, tortured, or killed anybody because they were atheists. From the Catholics of the Vendée to the Falun Gong in China right this minute, the history of the atheist left doing precisely that is as well-documented as it is horrifying. But the left will either simply deny this, or make bizarre assertions along the lines that Stalin had centuries-old churches dynamited or Mao had Tibetan monks tortured to death in prisons for some other, unspecified reason completely unrelated to anyone’s religious beliefs.)

(****And remember, if they are your theories, then it is your job to prove them, not ours to disprove them. And no, we’re not going to offer up our nation, our society, or our children’s cultural inheritance as a testing range for your unproven theories. If you want to test them, then move all of the believers in your theory to an island somewhere, and call us in a couple of centuries when you have some results that we can skeptically analyze.)

Liked it? Take a second to support Social Matter on Patreon!
View All

20 Comments

  1. Good thoughts. When you referenced the GPS story, did you mean to say “testing regimen?”

  2. I used to know a lot of engineers in China. Fun, smart guys but not prone to navel-gazing. If people were machines or motivated primarily by reason you might have a point. Envy-sloth-pride-lust-avarice-anger-gluttony and arête or excellence, grandeur, glory – these are (some of) the things that drive Europeans. I should quote Shakespeare at this point, I suppose, to round out the Western tragic view. So, how do technocrats quantify any of these mystical-genetically hardwired qualities?

    Better to keep “science” in its proper place, as a tool, subordinate to a study of Western Greats. Maybe like Bradbury’s Fahrenheit 451. A clandestine group who memorize and preserve European classics and tradition.

    1. But that’s precisely the point – it’s not about reasoning out nifty new theories, it’s about looking to the past to find what has provably worked in the real world. Unlike the “publish or perish” model of academia (which includes physicists), in engineering results are primary and theories secondary. Endlessly reinventing the wheel in an attempt to hit upon a perfect system is a terrible approach to problem-solving. In the “publish or perish” world, it’s a necessity, but in engineering, it’s a sure way to get fired.

      Engineers get paid to produce things that function in the real world, and nobody cares whether they function due to some novel, advanced theory, or function on a principle that the Athenians came up with. That’s what our model should be. It’s time to go back to using what we know works, with a few minor adjustments for some changed conditions.

      1. Sounds good to me. Too bad Jason Richwine wasn’t an engineer.

        “We” know what worked. A candidate for Sec. of Agriculture was asked by a Congressman why farmers in western Iowa were so productive. He said it’s because they’re German. Needless to say, he didn’t get nominated.

        1. Sorry. Meant “confirmed.” He was already nominated.

  3. Silicon valley makes plenty of trading algorithms and other expensive equipment that people cannot afford to break all the time, the point of making stuff that is “fail fast” is for epistemological reasons, that you cannot decide the direction of your companies growth because you have no idea what reality wants.

    1. Shipping a few reliable products along with a lot of unreliable ones does not make one a good model of engineering.

      If it was really not possible to have any idea what reality wants, best practices in business wouldn’t exist. There’s lots of collected wisdom out there about what to do to make a company work, and more importantly, what to do to make a company fail. None of this represents any guarantee of success, but if you want guarantees, this is the wrong universe to take up residence in.

  4. “Probably the earliest fly swatters were nothing more than some sort of striking surface attached to the end of a long stick.”

    — Jack Handy

  5. The general thrust of your article had good intensions. I adjust a little. The left has used “social engineering”, as they called it, exact, rigorous, binding and universal solutions to societies assumed or real problems. The problems of the lefts approach are

    1) Social world doesnt function on exact principles. There are motives, which are sometimes mutually or to the one way reinforcing; sometimes mutually or to the one way inhibiting; sometimes in jarring and vexing contradictions; sometimes almost totally separate and in no connection with each other; etc., like love, compassion, empathy, selfisness, desiring a partner, pity, need to belong socially, striving toward good social status, unconscious or conscious respect for authorities, contempt, anger, envy, schadenfreude, striving toward happiness, avoidance or neutralization of shame, guilt, confidence or over-confidence, jealousy, pleasure seeking, awe, disgust, feelings of trust, excitement, interest, fascination, etc. These become on and off, wane or become more intense, and their quality or nature changes or stays the same in response to changing or stable external situations, external incentives and disincentives, and inhibitory, preventive, exciting or inciting factors, which interact with internal states, memories, psychological predispositions, interpretive schemas, etc. Hence the social world is very complex and always in flux.

    2) There are no good solutions or desired end states, especially at the larger societal level. Best that can be achieved is a healthy and enduring balance between good and bad things; good incentive and disincentive structures, which tend to produce good solutions to personal and collective needs and problems, but which cannot be planned centrally beforehand; people who acquire virtuous habits, propensities, emotions, motives, etc. from living and growing in their social group, their community, by interacting with them; and people who have a good and tenacious ethnic, social and community identities and loyalties.

    3) The man will not be perfected, there will not be ideal man, brought on by intelligent and intensive social engineering. Man will always be flawed, imperfect, prone to sin, has weaknessess, etc. The man can learn to live with this and accept this reality, and have self-compassion and self-forgiveness. When he fails, he can repent when necessary, and always try again, again and again. Failing, erring and trying again is real life and good life.

    4) The intellectuals have at most less than 1% of the necessary knowledge in society. Knowledge in society and world is always radically dispersed. If intellectual wants to order what the whole society must do, ask him a simple question: “How do you produce and sell a pencil?” I.e. how do you plan and organize the search and mining operations for graphite, how do you construct the mining machines, how do you construct the machines that cut the woods, how do you construct the trucks, trains and ships that transport the raw materials and the end products, how do you make the machines that produce the pencils, how do you store, distribute, market and sell the pencils efficiently and economically, how do you organize and incentivice the organizations and work forces that do all of these, etc.? The same can be asked about all the necessary things in society. Soon you notice that the intellectual knows next to nothing about the necessary things in society. The distribution of power, possibilities, and resources should reflect this distribution of knowledge and capabilities, i.e. they should be decentralized. Intellectuals have limited use in society, and they should not be at the top power or deciding positions, despite the fact that they may have superior knowledge in some very narrow area. In good society there are variations in the way people live, work and organize, which reflect their different situations, propensities, ethnicities, capabilities, culture, opportunities, religious affiliations, etc. The good society is not coherent. The good society always includes at least some people that live differently that you or somebody else want them to live.

    1. Your ideas made sense until the “good society” stuff. No western civilization is going to work with large numbers of Muslims, blacks and feminists. Even in a police state, which could hardly be called civilized, the human rights racket would bankrupt it.

      1. IA,

        does it sound like I am advocating or accepting towards immigration or mass immigration? I have been consistently against immigration. There are diversity of ways of living among Europeans, Finns, Karelians (one of Finnish “tribes”), etc. I advocated acceptance toward this diversity. Although I admit that some non-Europeans in Europeans societies are likely to be necessary to increase our ethnocentrism and realism (and to decrease our propensity ideological fantasizing), too much non-European diversity has negative effects. Most European countries are long way past the necessary quantity of non-European diversity.

        1. You should say “ethnic European.” Otherwise “ethnicities” can be construed to include anybody.

  6. … and how radical is the distribution of knowledge? So radical that the person himself doesnt necessarily know beforehand how he will act in a certain situation, if he ends up in it. He may e.g. think or plan that he sells his car and saves his stereosystems, if he faces financial trouble, but it remains to be seen what he actually does, if that happens. In the actual situation he may sell the stereosystems and save the car, or sell them both, or save them both, sell or save some other combination of things or property. These alternative choices can feel to him very compelling. The choice depends on his thinking, emotions and their interaction with the situation, with its many aspects. This in addition that it is often hard for the person to predict to what kinds of situations he will end up, and this makes planning, decisions, knowledge etc beforehand even harder.

    1. My dad’s best friend served two tours as a platoon sergeant with the Marines in Vietnam. He said that when new guys came in, he’d always tell them that while the unpredictable could always happen, if your first instinct in a firefight was to follow your training, you were probably going to be okay. There’s a reason why drill sergeants beat that training into young Marines until it was second nature. They weren’t sadists – they understood that the Viet Cong weren’t going to take it easy on any of those soldiers once they got to Vietnam, so taking it easy on them in training was actually doing them a disservice. They needed that training – the tested, proven collected wisdom of years worth of combat experience – ingrained into them if they were going to have a chance of surviving.

      And it really was true. The guys who followed their training strongly tended to make it to the ends of their tours, while the guys who didn’t – who either shut down or decided to charge like they were in a John Wayne movie when the bullets started flying – strongly tended to end up dead quickly.

      The point is that the more that you ingrain tested, proven survival wisdom into a person – or a civilization – the more likely it is that they will survive. If, like the drill sergeant teaching his young Marines, we deeply want them to survive, then we will teach that survival wisdom to them even if we have to beat it into them.

      1. AntiDem,

        what you say is not in contradiction with what I said. Education, necessary basic and other knowledge is of course necessary, but once it has been acquired, it is applied, modified, supplemented, etc. in endless ways in practical life depending on the situations, including with additional studies. Before WW2 Finnish soldiers were taught to grease their guns. In winter wars freezing environments of – 40 – 50 C Finnish soldiers soon learned to polish all excess grease away, because the grease froze, and jammed the moving parts. Finnish guns remained operational in the harsh cold conditions, while many Russians found out too late that their guns grease had frozen and their guns didnt function. Also Finns improvised in the war fields and created e.g. “hell fields”. They hid barbed wire, barriers, slick surfaces, spikes etc. under the snow in certain arrangements, made the snow above them look natural and set hidden machine gun and other such emplacements in the surrounding area. When enemies walked into the area they were shot at. Panic ensued and they started to scatter into the surrounding area. They fell and/or become stuck to traps while fleeing, and it was then easy to kill them all with merciless fire.

        And so it goes in more peaceful situations too. Also traditions are not in contradiction with necessary changes.

  7. Of course we shouldn’t abandon the social sciences.

    We need victims to feed the mob.

  8. The phrase “Social Science” is an oxymoron. How can Science be social? Same with “Political Science”. This phrase was invented to give respectability to the agenda of “social engineering” by the vested interests, to manipulate and mold the minds of the masses, to create a certain perception conducive to “Cultural Marxism” etc. The only Science is Mathematics,Physics, and Chemistry in that order closely followed by Biology. Then there is such a thing as Arts and Humanities.

    1. The “social” in “social science” is a subject not an adjective. Obviously societies exist. Therefore obviously we may gain knowledge about them. Therefore obviously social science may exist. If you don’t like it, take it up with Aristotle.

  9. I’d suggest that instead of an engineering model for social science, we focus on a medicine model for science. Tho’ there is significant overlap in the philosophy of those two disciplines.

    1. But in general, Mr. Rustler is absolutely correct that a science of society is not at all like that of physics. And much harm comes from thinking that’s possible. Even geology, as Warg pointed out in our last podcast, is not like physics.

Comments are closed.