Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Leafsnail]

This is a criticism of cherry-picking/confirmation bias, not accepting anecdotes.

How do you prevent people from doing that if you're accepting random anecdotes as evidence of a wider trend?

[Bauglir]

And more than that, it tends toward a perverse excuse to stop thinking. To let the scientists do it for you, and just point to their results. Problem is, you've gotta get scientists somewhere - and if your scientific mindset gets shaped this way you choke your supply lines. Hell, even universities trend ever more toward teaching you material instead of teaching you how to think about it.

As somebody who fucking loves everything science has done for us, who admires objectivity and Truth for giving us ways to interact that don't rely on our gut impulses and opinions, who got a hard science education and a small stack of teaching experience, who spends almost all his time in front of a glowing box carved from pure logical thought and experiment (and also plastic, silicon, and various metals I guess), I find that direction to be absolutely, gob-smackingly, almost indescribably scary.

There's a need for humility, because recognizing we don't know everything is the only way to motivate us to learn more of it. And that means being willing to go, "Oh, hey, that's interesting, I'm going to have to think that over" and exerting ourselves to fit new information into a worldview that was never built to contain it. It means being willing to accept ideas that turn out to be wrong, too. Go out and test them, where they can be tested! That's how science is supposed to work, after all. But you're allowed to draw conclusions without the data, if you understand they're fallible. And more than that, you're allowed to draw conclusions with the data, as long as you understand that those are fallible too (just usually in different ways).

Nobody - nobody - is saying to disregard scientific evidence and go with your gut all the time. But to decide that nothing's true without the data to back it up is absurd - and that is precisely what it means to assert that accepting anecdotes at all leads to believing absolutely anything.

How do you prevent people from doing that if you're accepting random anecdotes as evidence of a wider trend?

Generally, you think about the anecdotes in the context of your experience with the world, determine whether any science exists that is relevant, weigh the relevant numbers alongside whatever anecdotes you happen to have, and then you make some decisions.

[SalmonGod]

You're horribly misunderstanding something. Statistics are gathered using a bunch of rules to eliminate bias that a simple "collection of anecdotes" won't really convey.

I could get a collection of anecdotes that says that every single person in America is actually a white libertarian between ages 18-25 and nobody would accept that as a proper result because it's demonstrably false.

Common sense is pretty much a meaningless phrase in general...

Ok, granted that I didn't specify it as a collection of anecdotes gathered according to a set of rules. 

But if "I'm poor and it makes me unhappy" as a simple individual statement is an anecdote, then there is nothing that fundamentally differentiates a statistical survey about income and happiness levels from a collection of anecdotes besides the rules regarding sample size and diversity for statistical significance.

[Leafsnail]

I'm sure rich people realize that, they just try to justify it by telling themselves that poor people deserve it (maybe relying on anecdotes about lazy or stupid poor people) - note that you can very easily examine the literature and find that poverty reduces your quality of life if you truly are only swayed by scientific evidence.  What this study found that was interesting and less obvious is that there's a cutoff point beyond which additional money does not make you happier.

[Frumple]

I... don't even want to touch the rest of the discussion, but insofar as statistics and happiness goes, SG, from what I recall the better ones don't actually ask questions like that. They ask about stuff like free time or hobbies, mental health issues, general physical health, time with friends/family, things along those lines. Behaviors we recognize happy people indulging in, or unhappy people experiencing.

Questions like income tied to something like, "Are you happy?" is honestly a kinda' terrible survey question in relation to good statistic gathering. That sort of thing is sometimes used, from what I understand, of course, but it's often more about framing (or perception) than it is actually getting numbers on happiness. Just a question like that prefacing a larger set of questions can (sometimes pretty strongly) influence what sort of answer you get, and people that actually know what the zog they're doing when it comes to statistic gathering know this.

I think there was someone on the forum (nenjin?) that actually had experience working with sociological research, though. They might have a better idea of the actual scutwork of it -- most of my understanding comes from interactions with folks doing the work, rather than doing the work itself.

[SalmonGod]

This is a criticism of cherry-picking/confirmation bias, not accepting anecdotes.

How do you prevent people from doing that if you're accepting random anecdotes as evidence of a wider trend?

That's down to self-awareness, which is a problem with how people react to information, science or not.  Many important subjects remain controversial because there is bogus paid-for science out there intentionally done just to feed people's confirmation bias.  It's an inescapable problem that can't be socially de-bugged by insisting people only trust scientific information.

Hell, this is a huuuuuuge problem in the business world, where employees complain about their working circumstances and decision-makers at the top insist that their reports don't show the numbers to reinforce those complaints, but those reports are designed to avoid reporting the problems employees complain about.  This is exactly the nightmare I'm dealing with at my job right now.

[Truean]

What SalmonGod said.... ^^^^ Yes.

The problem with anecdotal evidence isn't necessarily it isn't reliable. The problem is a.) People lack a proper bullshit detector, and b.) con men know it.

Can't study everything empirically, no matter how much we want that, "... more things in heaven and earth than are dreampt of in your philosophy my dear Horatio...." Pain, emotional, physical, 1-10 scale, is subjective. Confirmation bias, funding bias, Heteroscedasticity, Multimodal distribution, etc etc etc, screws up everything. Many researchers are never taught these things, countermeasures, and not to worship empiricism as a quasi religion.

Anecdotes have many problems, lack of counterfraud, confirmation bias, improper logic, emotional bias, etc, etc, etc. That said, housing bubble, corporate illegal corruption, etc, should never happen in empirical models. They do. That's even assuming the studies are done right and explained well. They aren't.

Sounds like blasphemy from me; it isn't. The problem is everybody has physics envy; nobody recognizes or talks about it. Everybody wants numbers on everything. Sometimes numbers rock, lots of medicine, engineering, etc. Anti vaccine propaganda is the prime offender here, but that alone doesn't invalidate anecdotes entirely. Anecdotes are important; it is sad nobody teaches how to evaluate them, reduce bias, emotion, fraud and illogical problems in them. Throwing numbers at problems, does not solve them, because that's not how humans work. Humans want quick, dirty, simple answers and they use numbers (often, bad or ugly) to say they're right as fast and easy as possible. Most people don't even know about problems in stats, deny they exist, don't know countermeasures, and won't do/pay for countermeasures in studies, but the faith is absolute.... This has been vastly abused, and that's why nobody trusts it anymore.... Abuse.... "When they own the information they can bend it all they want."

Bottom line, it's wayyyy too simplistic to say "anecdote bad; empirical studies good," because the world isn't simplistic.

People make decisions that effect other decisions based on illogical or semi logical cognitive operation functioning. Neoclassical Economics assumes perfect information, perfect price sensitivity (you'll change purchase decision over a penny), completely interchangeable everything, and perfect rationality, etc. It's all bullshit. That's not how the world or the people in it work. You shop around for a while, but you don't look at every choice on earth, so that's not perfect information. You won't drive all the way across town to save $0.01 a gallon on gas (you'll spend more money driving there, to say nothing of brand loyalty). Jesus, "perfect rationality," really? Forgetting human things like fraud, deceit, and marketing, we don't do things like that. We use the semi rational cognitive process called "satiation" in other words, "It's close enough." Stuff like that, etc. Good luck studying that empirically.

[Vector]

You're horribly misunderstanding something. Statistics are gathered using a bunch of rules to eliminate bias that a simple "collection of anecdotes" won't really convey.

I could get a collection of anecdotes that says that every single person in America is actually a white libertarian between ages 18-25 and nobody would accept that as a proper result because it's demonstrably false.

Common sense is pretty much a meaningless phrase in general...

I'd like to remind everyone that I have a degree in mathematics and specialized in measure theory (abstract probability theory). I have scientific and engineering training. I have worked in a job where my task was to create and present statistical research. That research is now published. It is science. I have attended rationality training sessions with the Center for Applied Rationality and have read many books on reducing bias and improving cognitive thinking skills.

My degree is from UC Berkeley, one of the top three mathematics programs in the united states, and the very best public university in the world. My undergraduate GPA was 3.7 and I was mentored by some of the university's top mathematical researchers. I completed 19 math courses, including two graduate courses, and I received a near-perfect (missed half a point on the writing section) on the GRE. An appeal to authority obviously does not mean that I am automatically correct, but I would hope that it would give you pause before you accuse me of fundamentally misunderstanding my chosen field of study.

I also have professional training in rhetoric, philosophy (specifically: ontology, i.e. What Is That Thing We Call the Truth, and epistemology--How Do We Find Out The Truth), literature, and art, and currently work as a literary researcher (Specifically: I work in verisimilitude and fictionality, and am currently working with Popper, i.e. the guy who invented Popperian Falsification). I would like to suggest to everyone here that perhaps these cultural artifacts that sustained us for millenia were there for a reason and still have something to teach us--and that something might be lost if we were to let go of these non-empirical modes of communicating our subjective experiences.

Common sense is usually where you point to people who aren't young white libertarians and say: "I can see that that person is a counterexample to your assertion." You should not have to point to a scientific study in order to figure that out.

I realize that I may sound something like the wackjobs you usually hear on the internet, but I have a fundamentally different position and the seeming kneejerk assumption that I'm a hostile, irrational, unenlightened person seems both a. off the mark to me and b. at this point, honestly disrespectful.

This is the point at which people usually say: "Oh, so you want me to bow down to you?!" No. I want you to look at yourself, and how you respect viewpoints that have Science written on them, and look at my arguments and see how I have a scientific background, and see that I am trying to convince you that having a scientific background does not mean that what you say is true, and see that you are disrespecting what I am saying despite my scientific background, and see that you are exercising my point exactly--except not willing to admit it because what I'm saying doesn't fit with your confirmation bias.

And then let's all laugh and have dumplings and green tea, because this is idiotic.

[SalmonGod]

I... don't even want to touch the rest of the discussion, but insofar as statistics and happiness goes, SG, from what I recall the better ones don't actually ask questions like that.

You're taking me too literally.  I never even said anything about what questions a survey would ask, anyway.  The point is, all this information is still being collected from the mouths or pencils of human beings, and the only thing that makes it more objective than a conversation is the scale and method.  The statistic has 10,000 conversations for me, and distills them down to a set of data points.  If I personally had 10,000 conversations and were an honest, self-reflecting person, I would still come away from it with those same data points.

If I encounter an anecdote that contradicts science, I'm very likely to disregard it.  If I encounter a large number of anecdotes that contradict science, I expect the science to explain that.  If I encounter anecdotes on subjects that I find little to no scientific material on, then I make sure I don't base my judgment on anecdotes from a single source.

Edit:

Ok... I guess I didn't read into it enough.  The guy is young.  Two years younger than me.  Looks to me like he hadn't been in his position of privilege for all that long.  In his case, it just took a little time to sink in how far he'd come, and that he was in a position to do something like this.  Very awesome.  Will follow Bauglir's lead and send a positive e-mail.

Anyway,  I doubt most people saw my observation that I edited in, because it was right at the bottom of the last page.  I don't think this was a case that boiled down to being convinced by science.  Although it's probably the reason the guy chose the target number that he did.

[Bauglir]

I want to quote you so bad right now, Truean, but instead I'll have to settle for a "Hell the fuck yes." Particularly your underlined bits.

On the plus side, I think I can quote Vector:

Common sense is usually where you point to people who aren't young white libertarians and say: "I can see that that person is a counterexample to your assertion." You should not have to point to a scientific study in order to figure that out.

Fuckin' this. It's not a matter of Science Is Bad, it's a matter of just not being blind.

But even more than that, this.

And then let's all laugh and have dumplings and green tea, because this is idiotic.

Though I have no dumplings. Only ice cream.

EDIT: I want to quote SalmonGod too, now, but if I keep doing that I'll never finish hitting Post, and adding to this one would really break up the flow.

[Leafsnail]

This is the point at which people usually say: "Oh, so you want me to bow down to you?!" No. I want you to look at yourself, and how you respect viewpoints that have Science written on them, and look at my arguments and see how I have a scientific background, and see that I am trying to convince you that having a scientific background does not mean that what you say is true, and see that you are disrespecting what I am saying despite my scientific background, and see that you are exercising my point exactly--except not willing to admit it because what I'm saying doesn't fit with your confirmation bias.

You are creating an absurd strawman.  Respecting the power of the scientific process does not mean that you believe anything that anyone with a science-related degree says - indeed, it should cause you to disregard misplaced appeals to authority such as the one you are making.  What it actually means is that you should respect scientific literature with clear methodology, results and analysis that has been reviewed by other experts in the field and ideally also replicated.

[Bauglir]

What it actually means is that you should respect scientific literature with clear methodology, results and analysis that has been reviewed by other experts in the field and ideally also replicated.

Yeah, I just want to say that this is true, but that's also not how people actually behave. For example:

Probably because once you buy into anecdotal evidence you can believe anything.

EDIT: To be clear, people often insert a superfluous "only" into your quote after "respect".

[Truean]

Bauglir, thank you. Take it as your own, but don't attribute it as from me to anyone. Say it is yours if and only if you believe it, or don't.

Leafsnail, Vector is expressly trying to avoid an appeal to authority. Her only point is she's not ignorant of many things on math. Also, "clear methodology," is rarer than many think. You have to be able to listen to what they're not saying, and that's really hard to ever explain. It's painted as too simple in an incredibly complex world.

It is so incredibly difficult to explain, that we criticize empiricism not for being a noble dream, with a noble aim, and a noble means, but for falling short of nobility and pretending not to be while claiming perfect noble title....

Why believe me. You do not know me. You have never seen me; you probably will never see me. I could lie to you, easily. Why wouldn't I? What reason do you have to trust one word you read from me?

Flaws. I have them. I admit them. I illustrate them. I am selfish. I can be wrong. I am so terribly flawed. I have made terrible mistakes I will regret my entire life and I can never atone for some of them, try as I might. Empiricism has assumed the fake corporate plastic picture perfect PR exterior. Anyone perfect must be lying, but even admitting your flaws can be abused to gain trust that is not earned..... Absolute rules, rarely apply absolutely.... This is not gravity we speak of....

[Glowcat]

Well, I'm not sure how much this contributes to the current(?) topic, but I think the problem with what is usually presented as anecdotal evidence isn't so much that it is a person's own experience but, similar to what SalmonGod said, it's a problem of narrowed evidence to the detriment of a larger and more accurate picture. What has happened vis a vis the "scientific community" and the silencing of minority voices has itself been gravely mistaken in a similar manner. Despite attempting to enforce rigor in empirical observations, it is unfortunately powerless against contamination of participants in the scientific project, effecting their ability to interpret, store, and validate data when it comes to matters that have a significant positional framing component. It is very easy for our underlying biases to direct what idealistically would be an accurate depiction of our shared material world. Worse yet, marked voices, such as those of feminists, are usually seen as being more biased by default by people within the system and its assumptions because they have rendered themselves disembodied minds within the ideas of pure reason when they are in fact not. They are tied to their body, and their society, and that mind isn't going to be able to escape that without significant self-critical awareness.

[Leafsnail]

EDIT: To be clear, people often insert a superfluous "only" into your quote after "respect".

You can't take a set of anecdotes as evidence of a wider trend though, precisely because the world is complex and things are not the same for everybody.  I think MetalSlimeHunt's quote is completely fair.

I think the vaccine/autism scare is the best example of this.  You can find plenty of "my child had the MMR vaccine and then they developed autism" anecdotes, and in the vast majority of cases those anecdotes are entirely true.  If you're prepared to accept anecdotal evidence then at this point you'd conclude that MMR vaccinations cause autism.  However, if you actually do the statistics you can find that rates of autism are no higher amongst vaccinated children than unvaccinated children, and that the association was only made because MMR vaccinations are generally given around the time when symptoms of autism begin to be seen.  There is no way you could find this out just by listening to anecdotes.