A Pre-emptive Dialectic (to weed out the weaker objections)

Ed:

Your dialectic solution only works on mechanistic problems. It will not work on this one.

http://yudkowsky.net/rational/bayes

1% of women at age forty who participate in routine screening have breast cancer. 80% of women with breast cancer will get positive mammographies. 9.6% of women without breast cancer will also get positive mammographies. A woman in this age group had a positive mammography in a routine screening. What is the probability that she actually has breast cancer?

Test your mechanistic solution on that truly Bayesian problem, Ed.

Post 201

Sunday, January 17, 2010 - 1:40pm

Robert,

Just because this problem doesn't lend itself to my solution doesn't mean that my solution is bad, wrong, false, or of low-value. It's actually just because this problem has such scarce information.

This problem, as stated, is simply a problem of counting frequencies and updating probabilities. Stated with such scarce information (and with roughly no integration at all), there is no other choice or solution -- other than to use a counting method and hope for the best.

But, and this is important, this is not how things are in real life. While the information given in this problem is appropriate on the population-wide scale of public health policy-making (and similar endeavors or projects), it wouldn't be enough information if it were your mom, wife, or daughter with the positive mammography.

To illuminate this, if it were your mom, wife, or daughter with a positive mammography -- would you be 7.8% worried? No. You would be approximately 100% worried until you utilized all-or-none reasoning such as my reasoning, in order to stem or quell your fears.

Using my reasoning then --as you would in real life -- you would progressively use methods that either identify (read: make the probability 100.0%) that your mom, wife, or daughter has breast cancer [so that it could be properly treated] or you would progressively use methods to show it to be increasingly impossible (probability = 0.0%) that your mom, wife, or daughter actually has breast cancer.

In this way, my reasoning is a more important and useful philosophical tool for living an individual life on earth (than is Bayes' theorem).

Ed
(Edited by Ed Thompson on 1/17, 6:45pm)

Post 202

Sunday, January 17, 2010 - 6:48pm

Addendum
Bayes Reasoning is something only proper for people or situations who are impoverished of relevant information or of sufficient integration of information.

In other words, it's a good tool for fools.

Post 203

Monday, January 18, 2010 - 3:42am

Ed,

As it happens, having incomplete information is a simple fact of life. (If you don't believe me, you can always go ask Godel, or Heisenberg.) For example, I lack information on where you are currently physically located, and you lack information on what percentage of broadcast Mythbusters episodes I've watched. If you define 'fool' as someone lacking any of the relevant information on a topic, as your post seems to suggest, then you have extended the definition of the word to include everyone, and robbed it of its usefulness in differentiating between fools and non-fools, making it meaningless. I don't think that was your intention, so I am therefore guessing that your post was based on incorrect assumptions about applying Bayesian reasoning.

Post 204

Monday, January 18, 2010 - 5:45am

Daniel,

I said it's good for fools, but that doesn't make it necessarily bad at all other times. I mentioned (demarcated) those times when it has an appropriate place (e.g., some population-wide policies).

Let me restate:
Fools will always "like" Bayes theorem, because they do not have to think about the nature and the relation of entities (i.e., they do not have to integrate, but merely have to count frequencies -- and pop them into an equation) in order to get "an answer" either way on something. Bayes will spit an answer back out at you regardless of the quality and scope of the information which you had put in. This is fine for fools.

Smart folks will sometimes "like" Bayes theorem, as when it applies to a context wherein very much needs to be done, but with relatively little "real-time" information -- such as during the outbreak of a communicable disease, working with the probability of infection or disease communication based on differing courses of action.

Ed

Post 205

Monday, January 18, 2010 - 5:54am

I guess you could say that fools can't see the difference between something like public health policy and their own, individual maximization of utility (or they don't want to see it -- because of the intellectual responsibility required).

It's these kind of people who look at the RDA for a vitamin and feel that that is what their own, individual body requires. Any mature nutrition scientist, however, will tell you that the RDA is not for individuals (as it only meets/exceeds the needs of the bottom 97-98% of folks).

The fool following the RDA may end up killing himself from doing that. It's a stretch, but it shows the inferiority of using broad, statistical tools in order to live your individual life (as opposed to a more comprehensive philosophy for living on earth, such as Objectivism).

Ed

Post 206

Monday, January 18, 2010 - 12:25pm

Ed,

It probably doesn't help that the main cheerleaders on this site for Bayes' Theorem are non-Objectivists, but this is one area where I'd strongly encourage you to reconsider the direction you've taken. Bayes' Theorem is so fundamental, elegant, and useful, not just for making population-wide policy, but for making personal choices. It is a key feature of so much rational decision-making and as such is entirely compatible with, and perhaps even vital for, Objectivism.

With respect, it seems to me you don't entirely understand the theorem or its applications. I would be willing to start a new thread to explain it such that you might appreciate its merit, or even discuss it with you directly via email.

Jordan

Post 207

Monday, January 18, 2010 - 12:55pm

Ed,

Bayesianism is about real life. Let's say your wish was granted and that 100% of women with breast cancer tested positive.

Using the Bayesian formula, the probability of a woman who tested positive actually having breast cancer is only 9.5%.

However, being told that the test is 100% accurate, a woman will be led to believe that her positive result means she definitely has breast cancer, when in fact the probability is far less. She would not be 9.5% worried, using your all-or-nothing method she would be 100% worried.

That fear is the reality doctors face every day when they have to confront their patients with test results.

The reality, Ed, is that drug companies use ignorance of the Bayesian theorem to promote their products. It is very common for a drug company salesman to claim that his drug has a 90% cure rate of all cases that tested positive for a disease. In fact, and in reality, applying the Bayesian formula may show that the vast majority of these positives were false. In that case, a 90% "cure" rate is definitely feasible because there was usually no disease present to cure.

Post 208

Monday, January 18, 2010 - 2:40pm

"That fear is the reality doctors face every day when they have to confront their patients with test results."

This is only a justifiable fear if you assume, as do elitist social-policy liberal fascists, that patients and laymen are ineducable fools, and that the possibility of a false positive is something that simply can't be explained to them. The elite always know that only they know better. The hapless and irrational layman can simply never understand. It's bullshit that under five minutes of actually talking to a patient as an adult can overcome.

Post 209

Monday, January 18, 2010 - 3:25pm

Ted,

85% of doctors, ignorant of, or even hostile toward, the Bayesian formula, are coming to patients with results that might worry or even frighten them.

If only it was as easy as talking for 5 minutes. It should be, but it's not. And it is not a matter of false positives. It could be that everybody in the civilized world knows about false positives. But how many are there? Only 15% of the doctors got the breast-cancer problem right because they chose intuition over counter-intuitive math. They chose the short-cut of intuition over the harder path of reasoning. In fact, there may be far, far more false positives than people intuitively realize because they mistakenly conclude that 80% accuracy means the rate of true positives will also be somewhere in that range, while ignoring the 1% frequency figure.

The last few days I emailed two doctor friends of mine with the problem. One of them got the answer right, 7.76%. The other one emailed back an answer of 70.4%. He didn't understand where he went wrong and had to ask me for the correct answer.

The Bayesian formula is designed to overcome a problem endemic to human reasoning. People tend to ignore the fact that only 1% of the women in the study actually had breast cancer. They ignore the fact that an 80% accuracy on diagnosing actual breast cancers says nothing by itself about the percentage of true positives for the group as a whole. They ignore the fact that if the frequency is 1% then the actual percentage of true positives will be far smaller than 70.4%.

Yet 85% of those well-educated doctors inevitably think the answer is somewhere between 70% and 80%. This despite the fact that they all took a class on statistics which included the Bayesian theorem. This despite the fact that doctor friend #2, who said the answer was 70.4%, had actually discussed this before with doctor friend #1 and took the same class in college.

When a doctor's intuition wins out over reasoning, a patient's psychological well-being loses.

(Edited by Robert Keele on 1/18, 3:28pm)

Post 210

Monday, January 18, 2010 - 3:36pm

Robert, I am glad for you that you have the unique ability to spot the shadows on the wall of the cave for what they are, and you have my full support for your campaign to be appointed philosopher king.

Post 211

Monday, January 18, 2010 - 3:50pm

Ted, if elected I will appoint you my heir-apparent to the throne.

Post 212

Monday, January 18, 2010 - 4:00pm

I should add, as a corollary, that if 99% of the women in the study are cancer negative, then this will cause the rate of false positives to skyrocket. In this study the probability of a false positive will be a whopping 92.24%.

Post 213

Monday, January 18, 2010 - 9:02pm

Jordan,

I would be willing to start a new thread to explain it such that you might appreciate its merit ...

Well, all I have to say about this is that:

The probability that I appreciate its merit, given that you start a new thread to explain it; is equal to the probability that you start a new thread to explain it, given that I appreciate its merit, multiplied by the prior probability that I appreciate its merit; all divided by the prior probability that you start a new thread to explain it -- which is unquantifiably helpful.

Ed

(Edited by Ed Thompson on 1/18, 9:03pm)

Post 214

Monday, January 18, 2010 - 10:05pm

Bayes' Theorem is not unlike the Gambler's Fallacy. Even those gamblers who know what the fallacy is tend to ignore it because they are emotionally hoping for a payoff based on false albeit seductive reasoning. This "reasoning" tells them that if a certain number has failed to come up over a certain length of time, then the odds of it coming up in the future increase. However, the odds of it occurring are no better on any particular occasion, the past does not predict the future, and all their expectation of outcomes does not improve the odds on iota.

To answer the question asked in the original post, I don't think that Objectivism and Bayes' Theorem are compatible. Because in order to accept Bayes' Theorem, or the Gambler's Fallacy, it is necessary to accept the fact that there is a subjective element to reasoning, and further, that this subjective side inevitably tries to influence decision-making. Powerless as it is against objective reasoning, it still attempts to assert its influence, seeking emotional inroads that temporarily short-circuit objective reasoning in order to gain more attention for its cause.

Post 215

Tuesday, January 19, 2010 - 6:20am

Robert Keele:

To answer the question asked in the original post, I don't think that Objectivism and Bayes' Theorem are compatible.

I'm curious about why you say this. Mathematically speaking, the formula is for deriving a probability from two or more populations. Some uses of Bayes' Theorem -- e.g. the probability of a hypothesis -- or interpretations (see here; the "objectivist view" there is not about Ayn Rand's Objectivism) are arguably incompatible.

Post 216

Tuesday, January 19, 2010 - 7:40am

Merlin,

I stated my reasons in the very post you're asking about, where I wrote:

Because in order to accept Bayes' Theorem, or the Gambler's Fallacy, it is necessary to accept the fact that there is a subjective element to reasoning, and further, that this subjective side inevitably tries to influence decision-making. Powerless as it is against objective reasoning, it still attempts to assert its influence, seeking emotional inroads that temporarily short-circuit objective reasoning in order to gain more attention for its cause.

Objectivism will never accept the idea of the inevitability of a side to reason that rejects objectivity. Objectivism will never accept the epistemic dualism of such a idea. Objectivism will never accept the idea that some "premises" simply cannot be changed, and that the tendency to fallacious reasoning is ever-present, despite being so easily exposed by the objective side of human nature.

But at least I am saying there is an objective side, and that it can avoid being influenced by the subjective side although with no hope of permanently vanquishing it.

Post 217

Tuesday, January 19, 2010 - 10:01am

I don't see that Bayes' Theorem necessarily entails a "subjective side."

Jordan

Post 218

Tuesday, January 19, 2010 - 10:17am

I don't see that Bayes' Theorem necessarily entails a "subjective side."

Ditto. It seems Mr. Keele is confounding Bayes' Theorem and Bayesian probability. Like the first link says:

Bayes theorem is valid in all common interpretations of probability, and is applicable in science and engineering. However, there is disagreement between frequentist and Bayesian and subjective and objective statisticians in regards to the proper implementation and extent of Bayes' theorem.

I think it is legitimate for a frequentist to view some probabilities as subjective. Suppose the following horse race case. Each horse has a track record, but none have ran against one another. Based on past races, what places the horse finished in such races, its times, how stiff the competition was, the track, the jockey and perhaps other factors, a person might assign odds to each horse winning the upcoming race. One might say, for example, that horse #3 has a 25% probability of winning. How should such a claim be interpreted? The frequency concept does not apply here. To base a probability claim on simply the past frequency of winning would be folly. All races are not equal. And only by coincidence would such frequencies for all the horses add to one. Such probability also cannot be an expectation about the frequency of winning an extended series of future races with the same horses, because the race will not be run again and again. What meaning can then be given to "probability" in such a situation? The only plausible answer is degree of confidence, a subjective view.

Clearly this is a much different situation that calculating the probability of getting two queens (or other combination) when drawing five cards from a standard 52-card deck.

(Edited by Merlin Jetton on 1/19, 10:50am)

Post 219

Tuesday, January 19, 2010 - 11:04am