Subject: Re: [extropy-chat] Re: Overconfidence and meta-rationality

From: [email-masked] ("Hal Finney")
Date: Thu, 10 Mar 2005 16:53:14 -0800 (PST)

I have enjoyed the debate between Eliezer and Robin (which I think
took place mostly on the wta-talk list), although I haven't been
able to follow it as closely as I'd like.  These long messages take
a lot of time to get through.  I wanted to make one comment on Eliezer's
posting:

> The modesty argument is important in one respect.  I agree that when two 
> humans disagree and have common knowledge of each other's opinion (or a 
> human approximation of common knowledge which does not require logical 
> omniscience), *at least one* human must be doing something wrong.

I'd put it a little differently.  There's nothing necessarily wrong
when two humans disagree and have common knowledge.  You have to add one
more ingredient.  The people have to both be rational and honest, and,
most importantly, they each have to believe that the other is rational
and honest (and, I think, this has to be common knowledge).

I would imagine that many cases of disagreement can be explained by
each party privately concluding that the other is being irrational.
They're just too polite to say so.  When they say, I guess we'll have
to agree to disagree, they mean, you're being unreasonable and I don't
want to argue with you any more because there's no point.

But actually, we can sharpen Aumann's result.  It doesn't require
assumptions about two people.  It is enough for one person to satisfy
the conditions.

Aumann basically says (neglecting the part about priors) that it is
impossible for a rational person to believe that he has a persistent
disagreement with another person whom he believes to be rational, where
the other person also believes the first person is rational.  Note that
this says nothing about what the second person actually believes.  It all
has to do with what the first person believes about the world.  It says
that a certain combination of beliefs is impossible for a rational person
to hold.

This perspective frees us from the competitive aspect of meta-rationality,
the "mine is bigger than yours" dynamic that sometimes arises in
discussions of this issue.  It's not a matter of one person being wrong
in a disagreement, or one person being more meta-rational than another.

Aumann is giving us a non-obvious piece of logic which we can follow in
our own thought processes, independent of what anyone else does.  I can't
fool myself into believing that I can agree to disagree with another
person, while respecting him as a rational and honest person who offers
the same respect towards me.  For me to hold this set of beliefs is a
logical contradiction.  That's the lesson I draw from this set of results.

In a way, then, Aumann can be read as giving you license to feel
contempt for others.  He's saying that it is mental hypocrisy (if that
means anything!) to try to adopt that generous and polite stance I just
described.  When we try to convince ourselves that we really believe
this noble fiction (that the other person is rational and honest), we are
lying to ourselves.  It's another case of self-deception.  The truth is,
we don't respect the other person as rational and honest.  If we did,
we wouldn't be ignoring his beliefs!  We think he's a fool or a knave.
Probably both.  We're not so damn nice as we try to pretend to be,
as we try to convince ourselves we are.

Hal
_______________________________________________
extropy-chat mailing list
[email-masked]
http://lists.extropy.org/mailman/listinfo/extropy-chat

---

From: "Eliezer S. Yudkowsky" <[email-masked]>
Date: Thu, 10 Mar 2005 17:53:38 -0800

> and honest (and, I think, this has to be common knowledge).

Suppose that one party is not rational.  I would call this, "doing 
something wrong".

We presume that both parties are honest because otherwise they are not 
"disagreeing" in the sense that I mean it, i.e., assigning different 
truth values.

Suppose that one party is rational and the other party fails to realize 
this.  Then the second party has failed to arrive to the correct answer 
on a question of fact.  Again, "something wrong".

If they just haven't figured it out yet, then they aren't necessarily 
doing something wrong.  They may be doing something right that takes 
time to accumulate evidence and computationally process it.  If they 
take too long or demand too much evidence, the beisutsukai sensei shouts 
"Too slow!" and whacks them on the head with a stick.  Speed matters in 
any martial art, including rationality, the martial art of thinking.

> the other person also believes the first person is rational.

I am not sure this is correct.  Maybe there is an extension of Aumann 
that says this, but it's not in Aumann's original result, which presumes 
rationality (i.e., irrationality is not considered as an option). 
Aumann-ish results, as far as I can see, tend to be about Bayesians 
treating other Bayesian's opinions as bearing a specific evidential 
relationship to the question at hand - the signals wouldn't have to be 
beliefs; they could as easily be flags that waved with a certain 
likelihood ratio.  In fact, what else is a Bayesian's belief, but a kind 
of cognitive flag that waves at only the right time?

> logical contradiction.  That's the lesson I draw from this set of results.

I agree.  But I regard rationality as quantitative, not qualitative.  I 
can respect an above-average rationalist while still occasionally 
wanting to shout "Too slow!" and whack him with a stick.

> as we try to convince ourselves we are.

Or you think that he's good and you're better.  That is also a 
self-consistent position to hold.  And if that is your position, you'd 
best not hide it from yourself - though based on my experience so far, I 
can't claim there will be any benefits forthcoming from public honesty 
about it.

-- 
Eliezer S. Yudkowsky                          http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence
_______________________________________________
extropy-chat mailing list
[email-masked]
http://lists.extropy.org/mailman/listinfo/extropy-chat

---

From: Russell Wallace <[email-masked]>
Date: Fri, 11 Mar 2005 01:54:45 +0000

On Thu, 10 Mar 2005 16:53:14 -0800 (PST), "Hal Finney" <[email-masked]> wrote:
> lying to ourselves.

For reference, anytime I say something to the effect of "I respect
that you're not being irrational, but I still disagree with you", or
something that implies such, I am not lying to either myself or the
person I'm talking to - I'm simply holding the position that Aumann
(if he's saying what you represent him as saying) is full of shit.

- Russell

---

From: Robin Hanson <[email-masked]>
Date: Sat, 12 Mar 2005 11:18:36 -0500

>that a certain combination of beliefs is impossible for a rational person

This is true, given that "rational" means "perfectly rational".  And of
course this is why Aumann's initial result seemed so easy to dismiss --
of course we are not sure that they are sure that we are sure that ...
we are both perfectly rational.  But more recent results are harder to
dismiss this way.  For example, this paper of mine,

<http://hanson.gmu.edu/disagree.pdf>For Bayesian Wannabes, Are 
Disagreements Not About Information?
<http://www.kluweronline.com/issn/0040-5833/>Theory and Decision 
54(2):105-123, March 2003,

allows for arbitrary deviations from perfect rationality.  It says that
if you both are reasonably confident that you both satisfy a few easy to
compute belief constraints, then you have to each believe that the other
person is biased in a certain direction *on average*, while each person
believes themselves to be unbiased.

>in a disagreement, or one person being more meta-rational than another.

Well, but it is in part.  Most dimensions of "bigger" are not directly
relevant. It is not directly about who knows more or thinks faster or
makes fewer cognitive errors.  It is only directly about calibration and
arrogance -  who on average better adjusts their estimates to account for
the fact that other people might be more right than they.  How various
mental "bigger" dimensions correlate with calibration remains an open
question to me.

Robin Hanson  [email-masked]  http://hanson.gmu.edu
Assistant Professor of Economics, George Mason University
MSN 1D3, Carow Hall, Fairfax VA 22030-4444
703-993-2326  FAX: 703-993-2323