Subject: Re: [extropy-chat] Re: Overconfidence and meta-rationality

From: [email-masked] ("Hal Finney")
Date: Mon, 21 Mar 2005 12:40:24 -0800 (PST)

Russell Wallace writes:
> On Mon, 21 Mar 2005 10:44:37 -0800 (PST), "Hal Finney" <[email-masked]> wrote:
> > You're both
> > born into the same world, products of the same evolutionary process;
> > you're exposed to different information, and ultimately your estimations
> > of probabilities are based on these causal factors.  Do you accept that
> > if you had been exposed to the experiences Elizer had, you would have
> > come up with his estimation of the probabilities, rather than yours?
>
> No. Our beliefs on things that aren't matters of proven fact depend
> not only on the domain-specific information we've received, but also
> on our genes and general experiences that form our personalities.

What about the thought experiment I offered to Eliezer: suppose you had
a duplicate made in a magical duplicating machine, and then the two
of you went off and studied some question for a year, independently.
Perhaps it's the one you disagreed upon with Eliezer.  When you return,
you each simultaneously announce your estimates of the probability for
the matter in question.

To make it dramatic, suppose that one of you decides that the probability
is low, say 1/10; while the other copy decides that the probability
is high, 9/10.  If you put yourself in that situation, would you find
that the mere knowledge that your copy got such a different result would
cause you to drastically revise your own opinion about the probability?
After all, you might as easily have been him, and if you had been
subjected to the same information as he was, you would presumably have
come up with essentially the same result.  In a symmetric situation like
this, doesn't it force you to view both probability estimates on equal
grounds, rather than favoring the one that you happen to hold?

Let's further impose the rather artificial restriction that all you can
do now is exchange your (updated) probability estimates with each other.
As you listen to the other copy's estimates your own probability estimates
may change, and you might expect his estimates to change as well.

What do you think about the claim that in this situation, it is
impossible, if you both view each other as honest, rational truth-seekers,
for you two to "agree to disagree"?  That you will both eventually
converge on a common estimate of the probability, even if you are not
allowed to explain the reasoning and information that led you to your
opinions?  Mere knowledge of the disagreement is enough to eliminate it,
according to the theorem.

By focusing on this rather artificial example, I am trying to eliminate
the variable you cited, the variation in genetics and prior experiences.
In this case, there will be variation due to experience over the course
of the experiment, but you will clearly start with the same set of biases
and beliefs ("priors", in Bayesian terms).  I wonder if you would then
find the no-disagreement argument persuasive.

Hal
_______________________________________________
extropy-chat mailing list
[email-masked]
http://lists.extropy.org/mailman/listinfo/extropy-chat

---

From: Russell Wallace <[email-masked]>
Date: Tue, 22 Mar 2005 00:58:21 +0000

> you each simultaneously announce your estimates of the probability for

Now, that does largely eliminate the other factors I mentioned, so
that the difference is indeed likely to be primarily in the
domain-specific knowledge each copy has access to.

Do you really mean "probability"? I suspect you don't, that you're
just following the habit some people have of saying "90% probability"
when you mean "I think so, though I'm not sure, but I actually have no
basis whatsoever for assigning a number to it". The Cyc project
started off doing this, but abandoned it, for good reason - it
obscures information. For example, it obscures the difference between
"I strongly believe X is probably true" and "I weakly believe X is
true".

But let's suppose I say what I would actually say, something like "I
think X is false, though I'm not certain" and my copy says "I think X
is true, but I'm not certain."

My reaction here will depend on what X is - in particular, if it
refers to the possibility of accomplishing something, then I will
reason as follows: "I believe X can't be done, my copy thinks it can.
What would make him think that? I'd say it's because he's figured or,
or been told about, a way to do it! Off the top of my head I can't
think of anything else that would change his mind like that. Alright,
then, I now believe it can be done."

So we won't converge on a middle ground - we'll both end up believing
X is true, but my conviction will be weaker than my copy's, because
it's based on more indirect reasoning.

> according to the theorem.

I don't agree there is any general rule to that effect, but you've
presented a situation where the prediction ends up being partly
correct, for reasons specific to the situation.

- Russell