The Analogy Problem on LLMs

Patrick Reany

3/24/2023

Definition: LLM refers to a Large Language Model machine, such as BingChat and ChatGPT.

My purpose in my investigations of these two chatbots is first for me to try to understand
how they think, and second, to try to help their users to be aware of pitfalls of
their performances.

On 3/22/2023 I compared how both BingChat and ChatGPT do on solving a proportion
problem. BingChat did it correctly, but ChatGPT got 'confused'. I'll clarify what I mean soon.

Patrick: 5 is to 3 as 11 is to what number?
ChatGPT: The relationship between 5 and 3 is that 5 is greater than 3 by a difference of 2. To
find the number that is related to 11 in the same way, we need to add 2 to 11. Therefore, the
answer is 13.

So, 5 : 3 :: 11 : 13.

Obviously, ChatGPT did not interpret what I gave it as a proportion problem. Apparently,
ChatGPT is thinking more broadly, such as to make the analogy:

medicine is to illness, as law is to anarchy,
which I barrowed from the site math.com. Such reasoning is used for SAT tests questions,
for example. By the way, the symbolic form of this analogy is:
medicine : illness :: law : anarchy.
This is why ChatGTP gave the answer in the form of "5 : 3 :: 11 : 13". ChatGPT interpreted
the problem I gave it as a difference problem, rather than as what most of us would interpret
it as, a division problem (i.e., a proportion problem).

On 3/23/2023, I asked ChatGTP to redo this problem, but only after I clued it in that it
is a proportion problem. Let's see what happened.

Patrick: The following problem is to be interpreted as a proportion problem
ChatGPT: Sure, I can help you with a proportion problem. Please provide the problem
statement and any additional information that may be helpful in solving the problem.
Patrick: 5 is to 3 as 11 is to what number?
ChatGPT: Great! This problem can be solved using proportion.

We know that 5 is to 3 as 11 is to some number, which we can represent as:

5/3 = 11/x

Here, x is the unknown number we need to find.

To solve for x, we can cross-multiply:

5x = 33

Then, we can isolate x by dividing both sides by 5:

x = 33/5

Therefore, the answer is:

11 is to 33/5 or 11:33/5

Okay, but I would have been happy if ChatGPT had stopped at "x = 33/5,"
but it's the 'right' answer this time.

I'm going to ask BingChat the same question.

Patrick: 5 is to 3 as 11 is to what number?
BC: To solve this analogy problem, we can use cross-multiplication.

5 is to 3 as 11 is to what number?

5/3 = 11/x

Cross-multiplying gives:

5x = 33

Dividing both sides by 5 gives:

x = 6.6

Therefore, the answer is 6.6.

This is correct, so I wanted to test it further by employing the abstract symbol \hbar,
which is a well-known symbol in physics.

------------------------------------

Patrick: 6 is to \hbar as 3 is to what number?
BC: This is another analogy problem.

6 is to \hbar as 3 is to what number?

Here, we can use cross-multiplication again.

6/\hbar = 3/x

Cross-multiplying gives:

6x = 3\hbar

Dividing both sides by 6 gives:

x = \hbar/2

Therefore, the answer is \hbar/2.

I get nervous in the service when BC describes what is clearly a proportion problem
as an 'analogy problem'. The problem is that by calling it an 'analogy problem',
one has set oneself up to apply the wrong interpretation, even if it is an
'analogy interpretation'. That's why ChatGPT misinterpreted the problem when it
wasn't specifically told to interpret it as a proportion problem. I suppose you
could say that this is a form of "machine overthinking."

Patrick: Do you know what I mean by \hbar?
BC: In physics, \hbar (pronounced "h-bar") is a fundamental constant that
appears in quantum mechanics. It is defined as:

\hbar = h/2π

where h is Planck's constant. The reduced Planck’s constant \hbar has dimensions
of angular momentum and is used to describe the quantization of angular momentum.

I hope this helps!

Patrick: Can you understand LaTeX coding?
BC: Yes, I can understand LaTeX coding. LaTeX is a document preparation system
that uses markup language to create documents. It is widely used in academia and
scientific research to write papers, reports, and books.
Patrick: Perhaps later.

BC: Sure, let me know if you have any questions later. I'm here to help!

So, that ends the chat at this point. I asked BC if it understood LaTeX because LaTeX
is the language I borrowed from to get the symbol '\hbar'. I challenged BingChat to
produce LaTeX code for certain mathematical markup purposes. That chat session
can be found soon in one of the other AI articles.