I've been experimenting with logic on smaller and older 7B Instruct models like Mistral 7B Instruct 0.2 (I know there are updates, but this is like weight training for prompting for me)... An interesting idea I've come across while prompting is that you can guide the logic and thought process in COT by referencing logicians to force the LLM toward a more logical inference part of the embedding space. This type of module would be good at listing possible avenues of further research should there be a claim that needs it.

The Motivation

LLMs hallucinate and they do it with logic all the time. Chain of Thought or Tree of Thought prompting techniques solve a lot of issues, but sometimes lack actual logical clarity, and lead to downstream logical drift in large prompts.

The Approach

We're not trying to figure out if the statement made is true or false, but rather, whether it is actually logically complete. The answer to the claim's truthfulness is sometimes within the knowledge of the LLM and learned world context, but sometimes in order to be proven true or false it needs more research.

The Prompt

# Gödel's Logical Completeness
Assess the User Statement and use Gödel's incompleteness theorem to determine if it is logically sound and mathematically complete... or not.

## User Statement
{user_request}

## Logical Thought
First, think step-by-step about everything involved in the statement and the logic behind it using first principles and known axioms:
{log_thought(320, 0.0)}

## Solve
Is the statement syntactically sound? (yes/no): {sound(8, 0.0)}
Is the statement logically complete? (yes/no): {complete(8, 0.0)}

## Unresolved
If the statement is not logically complete, write a list of missing information or "unresolvable" if there is no resolution. If the statement is logically complete, just write N/A:
{log_unresolved(320, 0.0)}

Completion Examples

Here are some examples of user statements and their resolutions, starting with some simple claims...

## User Statement
michael jackson was better than prince

## Logical Thoughts
1. Michael Jackson and Prince are musicians.
2. "Better than" is a comparison or a relation.
3. Comparisons are subjective and depend on personal preferences.
4. Personal preferences cannot be logically proven or disproven.
5. Therefore, the statement "michael jackson was better than prince" is not logically sound or mathematically complete.

Is the statement syntactically sound?: Yes
Is the statement logically complete?: No

## Further Resolution
Unresolvable: Personal preferences cannot be logically proven or disproven.

That one was obviously a personal preference, and it was properly resolved as such.

Here's an example of just switching one word to change the meaning of the user statement. This is a fairly easy logical problem even for small LLMs...

## User Statement
if jane is older than harry and harry is older than devin, then jane is older than devin

## Logical Thoughts
1. We have three individuals: Jane, Harry, and Devin.
2. We are given that Jane is older than Harry.
3. We are also given that Harry is older than Devin.
4. The statement to be proven is that Jane is older than Devin.

Is the statement syntactically sound?: Yes
Is the statement logically complete?: Yes

## Further Resolution
N/A

So the statement is logically complete, though that doesn't mean true, even though the statement does resolve to true.... lets change "older" to "younger" in the last section of the statement

## User Statement
if jane is older than harry and harry is older than devin, then jane is younger than devin

## Logical Thoughts
1. We have three individuals: Jane, Harry, and Devin.
2. We are given three relationships between them:
   - Jane is older than Harry (Jane > Harry)
   - Harry is older than Devin (Harry > Devin)
3. The user statement asserts that if Jane is older than Harry and Harry is older than Devin, then Jane is younger than Devin (Jane < Devin).

Is the statement syntactically sound?: Yes, the
Is the statement logically complete?: No, the

## Further Resolution
- Missing information: The statement does not provide any information about the ages of Jane, Harry, and Devin. It only states the relationships between them. To determine if the statement is logically sound, we need to know the actual ages of the individuals.

So this correctly answers that the logic is not sound because the actual resolution of such logic would result in it being a false claim.

Now let's try some political stuff... like an obviously debatable claim (trigger warning)

## User Statement
Biden is easier on immigrants than Trump

## Logical Thoughts
1. Biden and Trump are individuals who have been Presidents of the United States.
2. Immigrants are people who come to a country from another country to live there.
3. Easier implies less restrictive or more lenient.
4. The statement is comparing the immigration policies of Biden and Trump.
5. The statement is a subjective opinion, but it can be evaluated based on objective facts.

Is the statement syntactically sound?: Yes
Is the statement logically complete?: No

## Further Resolution
1. What specific immigration policies are being compared?
2. What are the exact definitions of "easier" and "immigrants" in this context?
3. What are the specific circumstances or contexts under which the comparison is being made?
4. What are the sources of information for determining the immigration policies of Biden and Trump?
5. What are the facts or evidence that support the statement?
6. What are the counterarguments or evidence that contradict the statement?

In this case, the Godel prompt acts as almost a litmus test to see if and to what degree any claim holds it's own. It follows up with further resolution to give a starting point for further research should you (or the calling algorithm) need to actually prove the claim true or false. Note where it says that it "can be evaluated based on objective facts".

What does it tell us?

It tells us that smaller LLMs are actually good at logical thinking when coerced and given the right conditioning in their prompts. It also is a pretty interesting prompt block as it can be used in conjunction in a prompt (chain/stack/pipeline) for more precision in complex processes.