When LLMs Try to Reason: Experiments in Text and Vision-Based Abstraction

fashions be taught to motive abstractly from just some examples? On this piece, I discover this query by testing each text-based (o3-mini) and image-capable (gpt-4.1) fashions on summary grid transformation duties. These experiments reveal the extent to which present fashions depend on sample matching, procedural heuristics, and symbolic shortcuts slightly than sturdy generalization. Even with multimodal inputs, reasoning usually breaks down within the face of refined abstraction. The outcomes supply a window into the present capabilities and limitations of in-context meta-learning with LLMs.

Introduction

Meta-learning, the flexibility of a system to discover ways to be taught, has historically been explored via gradient-based optimization, memory-augmented networks, or express activity embeddings. However with the rise of enormous language fashions (LLMs), significantly the o3 household with superior reasoning capabilities, a brand new query emerges: can we use LLMs themselves as meta-learners in task-based domains like ARC? The Abstraction and Reasoning Corpus (ARC), launched by François Chollet, is a benchmark explicitly designed to check broad generalization. It offers input-output transformation puzzles with minimal supervision, few examples per activity, and sometimes no shared surface-level construction throughout duties. In different phrases: a playground for meta-learning. To get an understanding of typical abstraction and reasoning duties, the reader can go to the ARC play page.

Knowledge and Setup

To discover whether or not LLMs like o3-mini can carry out meta-learning on summary reasoning duties, I used information from the ARC Prize 2025 Kaggle competition. The dataset repository could be discovered here (Apache 2.0 license). The dataset consists of input-output grid transformations that problem fashions to deduce summary guidelines from just some examples.

Every activity offers:

• A number of coaching examples (enter and output 2D grids)
• A single take a look at enter grid for which the mannequin should predict the corresponding output

A second dataset offers the answer grids for every of the take a look at enter grids. Right here’s a simplified instance of the info format:

# coaching examples - dictionary of dictionaries. 
# Right here is an extracted activity
{'prepare': [{'input': [[6, 6, 0], [6, 0, 0], [0, 6, 6]],
   'output': [[6, 6, 0, 6, 6, 0, 0, 0, 0],
    [6, 0, 0, 6, 0, 0, 0, 0, 0],
    [0, 6, 6, 0, 6, 6, 0, 0, 0],
    [6, 6, 0, 0, 0, 0, 0, 0, 0],
    [6, 0, 0, 0, 0, 0, 0, 0, 0],
    [0, 6, 6, 0, 0, 0, 0, 0, 0],
    [0, 0, 0, 6, 6, 0, 6, 6, 0],
    [0, 0, 0, 6, 0, 0, 6, 0, 0],
    [0, 0, 0, 0, 6, 6, 0, 6, 6]]},
  {'enter': [[4, 0, 4], [0, 0, 0], [0, 4, 0]],
   'output': [[4, 0, 4, 0, 0, 0, 4, 0, 4],
    [0, 0, 0, 0, 0, 0, 0, 0, 0],
    [0, 4, 0, 0, 0, 0, 0, 4, 0],
    [0, 0, 0, 0, 0, 0, 0, 0, 0],
    [0, 0, 0, 0, 0, 0, 0, 0, 0],
    [0, 0, 0, 0, 0, 0, 0, 0, 0],
    [0, 0, 0, 4, 0, 4, 0, 0, 0],
    [0, 0, 0, 0, 0, 0, 0, 0, 0],
    [0, 0, 0, 0, 4, 0, 0, 0, 0]]},...,
   'take a look at': [{'input': [[7, 0, 7], [7, 0, 7], [7, 7, 0]]}]
}

# instance of resolution to check enter grid - dictionary of lists
# Right here is the extracted resolution for the one take a look at enter grid above
[[[3, 2, 3, 2, 3, 2],
  [7, 8, 7, 8, 7, 8],
  [2, 3, 2, 3, 2, 3],
  [8, 7, 8, 7, 8, 7],
  [3, 2, 3, 2, 3, 2],
  [7, 8, 7, 8, 7, 8]]]

Every grid is a 2D array of integers from 0–9, representing coloured pixels. Grids have various sizes and a grid transformation might also carry a dimension change from the enter to the output grid. To visualise the arrays, I used a customized colormap with matplotlib:

from matplotlib import colours
cmap = colours.ListedColormap([

    '#8B00FF',  # Violet
    '#4B0082',  # Indigo
    '#0000FF',  # Blue
    '#FFFF00',  # Yellow
    '#00FF00',  # Green
    '#FF7F00',  # Orange
    '#FF0000',  # Red
    '#964B00',  # Golden
    '#000000',  # Black
    '#FFFFFF',  # White
])
norm = colours.Normalize(vmin=0, vmax=9)

# Operate to visualise an array
def visualize_matrix(matrix, title='', cmap=cmap, norm=norm):
    plt.imshow(matrix, cmap=cmap, norm=norm)
    plt.title(title)
    plt.axis('off')  # Take away axes
    plt.present()

For mannequin interplay, I used OpenAI’s o3-mini mannequin by way of LangChain. In a while, we may also use gpt-4.1:

from langchain_openai import ChatOpenAI
import getpass
import os

# Immediate for a secret enter
openai_key = getpass.getpass("Enter your OpenAI API key: ")

os.environ["OPENAI_API_KEY"] = openai_key

AGENT_MODEL = "o3-mini"  # reasoning mannequin, https://platform.openai.com/docs/fashions
AGENT_LLM = ChatOpenAI(mannequin=AGENT_MODEL) 
# AGENT_LLM = ChatOpenAI(mannequin=AGENT_MODEL, reasoning_effort='low')

To deal with LLM responses, particularly when the mannequin returns a predicted output grid as Python code inside triple backticks, I wrote a utility:

import re, ast

def extract_python_code(response_string):
    match = re.search(r"```pythons*(.*?)```", response_string, re.DOTALL)
    if match:
        return ast.literal_eval(match.group(1).strip())
    return None

This setup allowed me to construction a full reasoning loop: immediate the mannequin with few-shot examples, extract and apply a generated algorithm, assess its efficiency on new take a look at inputs and at last use the evaluation to enhance the algorithm.

Testing Reasoning with o3-mini

To guage whether or not LLMs can “meta-learn” on summary reasoning duties, I examined the o3-mini mannequin utilizing a closed-loop reasoning setup impressed by how people may method few-shot duties. For every ARC problem, I supplied the mannequin with a handful of demonstration input-output grid pairs and requested it to derive a single reusable algorithm.

I outlined a sequence of prompts utilizing LangChain’s ChatPromptTemplate to simulate reasoning, software, evaluation, and refinement. The method mimics an inside coaching loop with restricted supervision:

• PROMPT_REASON: The mannequin is given coaching examples and requested to deduce a common algorithm in pseudocode.
• PROMPT_SOLVE: The generated algorithm is utilized to new inputs (each coaching and take a look at).
• PROMPT_ASSESS: When the algorithm fails, the mannequin receives suggestions evaluating its predicted vs. anticipated outputs.
• PROMPT_SUMMARIZE_FEEDBACK: The mannequin summarizes cumulative suggestions from failed makes an attempt to iteratively refine its method.

from langchain_core.prompts import ChatPromptTemplate

PROMPT_REASON = ChatPromptTemplate.from_messages(
    [
        (
            "system", 
            "You are an expert in solving abstract reasoning tasks. "
            "You will be given several demonstration input-output pairs of 2D arrays. "
            "Your goal is to develop a single algorithm that maps each input array to its corresponding output array.nn"
            
            "Each input and output is a 2-dimensional array of integers between 0 and 9. "
            "Solving the task involves:n"
            "- Analyzing the demonstration pairsn"
            "- Identifying abstract patterns or transformationsn"
            "- Formulating a general rule or algorithm that works across all examplesn"
            "- Producing pseudocode that implements the rulenn"
            
            "If prior attempts were made, you will also receive feedback summarizing what went wrong. "
            "Carefully use this feedback to improve your solution.nn"
            
            "Return only the updated algorithm as pseudocode. Do not describe or explain it.nn"
            "### Feedback (summary of previous attempts):n{attempt_history}nn"
            "### Demonstration Pairs:n{train_pairs}n"
        ),
        (
            "ai", 
            "Answer:"
        )
    ]
)

PROMPT_SOLVE = ChatPromptTemplate.from_messages(
    [
        (
            "system", 
            "You are an expert in abstract reasoning. "
            "Previously, you analyzed demonstration input-output pairs and developed an algorithm "
            "to transform input arrays into output arrays.nn"
            
            "Now, use that algorithm to generate an output array for a new, unseen input array.nn"
            
            "Only return the output array, formatted as valid Python code within a code block. "
            "For example:n```pythonn[[2, 3], [5, 6]]n```n"
            
            "### Developed algorithm:n{reasoning_template}nn"
            "### New enter array:n{test_input}n"
        ),
        (
            "ai",
            "Reply:"
        )
    ]
)

PROMPT_ASSESS = ChatPromptTemplate.from_messages(
    [
        (
            "system", 
            "You are an expert in abstract reasoning. "
            "A solution array was generated by applying the algorithm to the input array. "
            "Compare the generated solution to the actual target output. "
            "Analyze why the two arrays differ, and provide **clear and concise feedback** on how to improve the algorithm.nn"
            
            "Only return your feedback-do not repeat the arrays or algorithm.nn"
            
            "### Algorithm:n{reasoning_template}nn"
            "### Input array:n{test_input}nn"
            "### Solution array (generated by algorithm):n{solved_test_output}nn"
            "### Target output array:n{test_output}n"
        ),
        (
            "ai",
            "Answer:"
        )
    ]
)

PROMPT_SUMMARIZE_FEEDBACK = ChatPromptTemplate.from_messages(
    [
        (
            "system", 
            "You are an expert in summarizing feedback on algorithm development. "
            "You will be given a history of past attempts, each containing an algorithm and feedback about its performance.nn"
            
            "Your goal is to produce a **concise summary** of the most important lessons learned-"
            "focusing on how the algorithm should be improved and what mistakes should be avoided in future versions.nn"
            
            "Return only the feedback summary. Do not repeat the original attempts or feedback.nn"
            
            "### Attempt History:n{attempt_history}n"
        ),
        (
            "ai",
            "Answer:"
        )
    ]
)

These prompts are linked right into a easy LangChain pipeline:

reasoning_chain = PROMPT_REASON | AGENT_LLM
solve_chain = PROMPT_SOLVE | AGENT_LLM 
assess_chain = PROMPT_ASSESS | AGENT_LLM 
summarize_feedback_chain = PROMPT_SUMMARIZE_FEEDBACK | AGENT_LLM

For every ARC problem:

• The mannequin receives the demonstration pairs and prior suggestions;
• The mannequin generates a brand new algorithm in pseudocode (reasoning_template);
• The algorithm is examined on all of the demonstrations;
• If it fails, the mannequin: receives detailed suggestions on mismatched predictions; summarizes errors throughout makes an attempt; refines the following model of the algorithm;
• As soon as the mannequin will get all demonstrations right, I take a look at it on the unseen take a look at enter.

This course of repeats for as much as a max variety of makes an attempt per problem. A profitable algorithm generalizes throughout the supplied examples and applies accurately to the withheld take a look at case. This setup checks whether or not the mannequin can extract summary patterns, enhance its reasoning over time, and generalize from only a few examples.

reasoning_templates = {}

for i, id in enumerate(id_train_challenges):
    print(f"Coaching on problem {i} ID: {id}")
    train_pairs = train_challenges[id]['train']
    test_input = train_challenges[id]['test'][0]['input'] # solely choose the primary take a look at enter 
    test_output = train_sols[id][0] # solely choose the primary take a look at output
    train_pairs_str = ''
    for i, train_pair in enumerate(train_pairs):
        train_pairs_str += f"Demonstration pair {i+1}:n enter grid: {train_pair['input']} n output grid: {train_pair['output']}n"
    train_pairs_str = train_pairs_str.strip()

    # preserve making an attempt till you determine the right way to resolve the problem
    right_wrong = "incorrect"
    # Begin with an empty reasoning template, which shall be refined over time
    reasoning_template = '' 
    okay = 1
    max_attempts = 5
    attempt_history = []
    attempt_history_summary = ''
    whereas right_wrong == "incorrect":
        print(f"Try {okay} to resolve the problem...")

        # Construct the reasoning message with the present reasoning template and try historical past
        # This message shall be used to generate a brand new reasoning template
        reason_message = {
            "train_pairs": train_pairs_str,
            "attempt_history": attempt_history_summary,  
        }
        res = reasoning_chain.invoke(reason_message)
        reasoning_template = res.content material

        # Assess reasoning template
        wrong_pairs = []
        for train_pair in train_pairs:
            demo_input = train_pair['input']
            demo_output = train_pair['output']
            # Check the reasoning template on the demonstration pair
            test_message = {
                "test_input": demo_input,
                "reasoning_template": reasoning_template,
            }
            res = solve_chain.invoke(test_message)
            solved_demo_output = extract_python_code(res.content material)            
            # Examine the output with the demonstration output
            if solved_demo_output != demo_output:
                wrong_pairs.append((demo_input, demo_output, solved_demo_output))

        if len(wrong_pairs) > 0:
            right_wrong = 'incorrect'
            print(f"Reasoning template failed on {len(wrong_pairs)} demonstration pairs.")

            if okay >= max_attempts:
                print(f"Max makes an attempt reached ({max_attempts}). Stopping for problem {id}.")
                reasoning_templates[id] = ''
                break

            print("Assessing the reasoning template...")
            assessment_res = f'Algorithm failed on {len(wrong_pairs)} demonstration pairs. Right here is the suggestions:n'
            for demo_input, demo_output, solved_demo_output in wrong_pairs:
                assess_chain_message = {
                    "reasoning_template": reasoning_template,
                    "test_input": demo_input,
                    "solved_test_output": solved_demo_output,
                    "test_output": demo_output,
                }
                res = assess_chain.invoke(assess_chain_message)
                assessment_res += f" - From enter {demo_input} to output {demo_output}, your resolution was {solved_demo_output}: {res.content material.strip()}n"

            attempt_history.append({
                "try": okay,
                "reasoning_template": reasoning_template,
                "suggestions": assessment_res
            })

            summary_message = {
                "attempt_history": attempt_history,
            }
            summary_res = summarize_feedback_chain.invoke(summary_message)
            attempt_history_summary = summary_res.content material.strip()
        else:
            print("Resolution is right!")
            right_wrong = "right"
            reasoning_templates[id] = reasoning_template

            # take a look at it towards the take a look at enter/ output .... however don't give suggestions (that is alleged to be unknown)
            test_message = {
                "test_input": test_input,
                "reasoning_template": reasoning_template,
            }
            res = solve_chain.invoke(test_message)
            solved_test_output = extract_python_code(res.content material)
            if test_output != solved_test_output:
                print(f"Check output doesn't match the true output for problem {id}.")
            else:
                print(f"Check output matches the true output for problem {id}.")
                #visualize_matrix(test_input, "Enter grid")
                #visualize_matrix(test_output, "True output")
                #visualize_matrix(solved_test_output, "Check Output")

            print("-" * 40)  # Separator between entries

        okay += 1

Outcomes: When Reasoning Works

In some instances, o3-mini was in a position to accurately infer a generalizable algorithm from just some input-output demonstrations. One such instance concerned producing a patterned tiling based mostly on a small 2×2 enter grid.

After only one try, the mannequin converged on the next pseudocode:

BEGIN  
  Let enter be a 2x2 grid, the place:
    enter[0] = [a, b]
    enter[1] = [c, d]
  
  Initialize output as an empty listing.

  FOR every row index r from 0 to five DO:
    Let original_row ← enter[r mod 2]
    
    IF (FLOOR(r / 2)) mod 2 = 1 THEN
      Let base_row ← REVERSE(original_row)
    ELSE
      Let base_row ← original_row
    ENDIF

    Initialize new_row as an empty listing.
    FOR repeat from 1 to three DO:
      Append all components of base_row to new_row.
    ENDFOR

    Append new_row to output.
  ENDFOR

  RETURN output
END

Right here is the anticipated resolution (true output) and the answer from the mannequin pseudocode (take a look at output).

This algorithm demonstrates a number of notable reasoning capabilities:

• Sample abstraction: the mannequin inferred a repeating tiling sample from restricted information;
• Modulo logic: it launched modulo-based indexing (r mod 2 and (r // 2) mod 2) to alternate row habits, mimicking visible mirroring;
• Grid building: the answer scales up the two×2 enter into a bigger 6×6 grid via repeated copying and reversal;

The truth that the mannequin found this construction with out hardcoded guidelines suggests it was participating in a type of algorithm synthesis, guided by few-shot abstraction.

Right here is one other profitable instance. 

Once more, after only one try, the mannequin converged on the next pseudocode:

BEGIN
  Let N = 3
  Create output as a 2D array of dimension (N×N) × (N×N), stuffed with 0

  FOR every row r from 0 to N–1:
    FOR every column c from 0 to N–1:
      IF enter[r][c] ≠ 0 THEN
        FOR every i from 0 to N–1:
          FOR every j from 0 to N–1:
            Set output[(r * N) + i][(c * N) + j] = enter[i][j]
  RETURN output
END

Right here is the anticipated resolution and the answer from the mannequin pseudocode.

This algorithm successfully tiles the total enter grid into the output grid at each place the place the enter cell is non-zero. The tiling is aligned such that every copy of the unique grid is positioned at an offset decided by (r * N, c * N) —  the scaled-up coordinates of the non-zero enter cell.

What’s spectacular right here is that the mannequin:

• Learns conditional placement. It solely pastes the enter the place the enter worth is non-zero;
• Makes use of coordinate arithmetic to scale placement areas, displaying spatial understanding;
• Treats the enter as each management logic and content material, combining format detection with repetition.

When Textual content-Based mostly Reasoning Isn’t Sufficient

Within the activity under, the mannequin was given a small variety of demonstration input-output grid pairs and requested to deduce the transformation rule. 

The text-based mannequin (o3-mini) produced an in depth pseudocode resolution, structured, believable, and internally constant:

BEGIN
  Let unique ← enter grid  
  Let output ← deep copy of unique
  Let R ← variety of rows in unique  
  Let C ← variety of columns in unique

  // Compute ring index for each cell that's a part of a non-zero area.
  // A cell's ring index is outlined as:
  //   - 0 if the cell is on the boundary of the grid OR if at the least one in all its 4-neighbors is 0
  //   - In any other case, 1 + min(ring index of its 4-neighbors)

  Create grid ring of dimension R × C, stuffed with −1

  For every cell (r, c) in unique:
    If unique[r][c] ≠ 0 then
      If r = 0 OR c = 0 OR r = R−1 OR c = C−1 then
        ring[r][c] ← 0
      Else if any 4-neighbor (up, down, left, proper) of (r, c) has worth 0 then
        ring[r][c] ← 0

  modified ← true
  Whereas modified is true:
    Set modified ← false
    For every cell (r, c) in unique such that unique[r][c] ≠ 0 and (ring[r][c] < 0 or not finalized):
      Let candidates ← listing of ring indices from any 4-neighbor (r′, c′) with a non-negative ring worth
      If candidates is non-empty then
        newRing ← min(candidates) + 1
        If ring[r][c] < 0 OR newRing < ring[r][c] then
          ring[r][c] ← newRing
          modified ← true

  // Course of propagation from outer ring inward.
  // For every cell with ring index ≥ 1, replace its worth based mostly on a neighbor with ring index one much less.
  // Choice is deterministic: "up, left, down, proper"

  Let maxRing ← most worth in ring

  For layer from 1 to maxRing:
    For every cell (r, c) such that ring[r][c] = layer:
      For course in [(−1,0), (0,−1), (1,0), (0,1)]:
        nr ← r + course.row
        nc ← c + course.col
        If (nr, nc) is in bounds AND ring[nr][nc] = layer − 1 then
          Let base ← output[nr][nc]
          If base = 2 then
            newValue ← 8
          Else if base mod 2 = 0 then
            newValue ← base + 2
          Else // base is odd
            newValue ← base + 1
          output[r][c] ← newValue
          Escape of neighbor loop

Return output
END

At first look, this appears to be like like a classy technique: compute a hoop index for every non-zero area, then diffuse values inward utilizing deterministic guidelines. It even consists of replace logic:

• If the “base” cell is 2 → assign 8
• If base is even → add 2
• If base is odd → add 1

However this whole algorithm, nevertheless coherent, is misguided. It utterly fails to match the true underlying transformation proven within the demonstrations.

As an alternative of reasoning about blue-bordered areas and their nested construction, the mannequin generated a generic flood-fill algorithm based mostly on distance from edge and adjacency to zeros. Even the suggestions retains refining the procedural method recognized earlier:

['Key lessons are to: • Precisely compute the ring index so that only',
 'true boundary (or external zero) cells get index 0, ensuring that',
 'inner cells receive higher indices for proper propagation. • Use a',
 'reliable, consistent method for selecting the "base" value for',
 'updates-ideally by considering all adjacent lower-ring cells or using',
 'a deterministic order-and use an immutable copy of the original grid',
 'for these lookups. • Apply the parity‐based update rules correctly so',
 'that cells with ring index ≥ 1 get the specified value increments',
 '(especially the special case when the base is 2) rather than remaining',
 'unchanged. • Ensure that the update logic cascades inward, allowing',
 'inner cells to correctly inherit and build upon values from outer',
 'rings.']

So what went improper?

• Topological, not visible. The mannequin centered on connectivity and edge proximity, ignoring the visually outlined areas.
• Procedural, not inferential. The logic was inflexible and hand-crafted, not derived from patterns within the examples.
• Demonstration-agnostic. There’s no signal the mannequin meaningfully integrated the few-shot examples. It seemingly defaulted to a well-known sample — spatial progress utilizing layers.

This isn’t stunning. Textual content-only LLMs haven’t any visible grounding. They tokenize the grid as symbolic enter — rows of digits, not enclosed figures or nested patterns. Consequently, their inductive biases lean towards symbolic or graph-like algorithms, not perceptual abstractions.

On this case, the mannequin fell into a typical entice: producing one thing plausible-sounding however improper. It produced a spatial propagation scheme which may work for a diffusion activity however not the one at hand. This highlights a key weak point in text-based few-shot prompting for summary visible reasoning: the mannequin’s “reasoning” is disconnected from perceptual understanding. It invents algorithms based mostly on inside priors, not exterior cues.

When Reasoning Fails: Additionally Picture Fashions Aren’t Magic

To enhance generalization, I transitioned from purely text-based reasoning to image-based prompting, leveraging GPT-4.1’s multimodal capabilities via LangChain. This setup encoded input-output grid examples as base64 pictures, which had been offered alongside a pure language immediate describing the duty.

from langchain_core.messages import HumanMessage

import io
import base64

AGENT_MODEL = "gpt-4.1"

# Immediate for picture based mostly reasoning
PROMPT_REASON_IMG = """You're an skilled at fixing summary reasoning duties.

These are distinctive reasoning duties with restricted examples. You're given demonstration input-output 2D grids. 
The colormap used is as follows:

{{
    'Violet': 0,
    'Indigo': 1,
    'Blue': 2,
    'Yellow': 3,
    'Inexperienced': 4,
    'Orange': 5,
    'Crimson': 6,
    'Golden': 7,
    'Black': 8,
    'White': 9
}}

Your aim is to develop a single algorithm that maps every enter grid to its corresponding output grid.

A profitable resolution entails:
- Analyzing the demonstration examples rigorously
- Figuring out underlying visible or spatial patterns
- Formulating a common transformation rule
- Translating this rule into clear pseudocode

If this isn't your first try, a abstract of earlier suggestions can be supplied. Overview it rigorously and incorporate it to enhance your resolution.

Check your algorithm towards the demonstrations to make sure it really works.

Return **solely the algorithm pseudocode**, formatted as plain textual content. Don't clarify it or add further commentary.
"""

# In case your array is 10x10 and also you need every cell to be 20x20 pixels (cell_px), the picture shall be 200x200 pixels.
# Convert matrix into picture
def visualize_grid_fig(matrix, cmap=cmap, norm=norm, cell_px=20, present=False):
    if sort(matrix) isn't np.ndarray:
        matrix = np.array(matrix)
    h, w = matrix.form[:2]
    figsize = (w * cell_px / 100, h * cell_px / 100)  # inches
    fig, ax = plt.subplots(figsize=figsize)
    ax.imshow(matrix, cmap=cmap, norm=norm)
    ax.axis('off')
    if present:
        plt.present()
    else:
        plt.shut(fig)
    return fig

# encode picture for mannequin
def fig_to_base64(fig, dpi=100):
    buf = io.BytesIO()
    fig.savefig(buf, format='png', dpi=dpi, bbox_inches='tight')
    buf.search(0)
    img_base64 = base64.b64encode(buf.learn()).decode('utf-8')
    buf.shut()
    return img_base64

# Within the loop substitute reasoning code with this
# reasoning with pictures
reason_message = [{"type": "text", "text": PROMPT_REASON_IMG}]
for i, instance in enumerate(train_pairs):
    #fig_in = visualize_grid_fig(instance['input'], cmap, norm)
    #fig_out = visualize_grid_fig(instance['output'], cmap, norm)
    fig_in = visualize_grid_fig(instance['input'], )
    fig_out = visualize_grid_fig(instance['output'], )
    fig_in = fig_to_base64(fig_in)
    fig_out = fig_to_base64(fig_out)
    reason_message.append({"sort": "textual content", "textual content": f"### Enter grid {i+1}:"})
    reason_message.append({"sort": "image_url", "image_url": {"url": f"information:picture/jpeg;base64,{fig_in}"}})
    reason_message.append({"sort": "textual content", "textual content": f"### Output grid {i+1}:"})
    reason_message.append({"sort": "image_url", "image_url": {"url": f"information:picture/jpeg;base64,{fig_out}"}})
reason_message.append({"sort": "textual content", "textual content": f"### Suggestions (abstract of earlier makes an attempt): {attempt_history_summary}"})
reason_message = HumanMessage(content material=reason_message)
res = AGENT_LLM.invoke([reason_message])
reasoning_template = res.content material

The ensuing pseudocode marked a transparent step ahead in expressiveness. The mannequin was in a position to:

• Detect blue-bordered squares utilizing visible options slightly than purely symbolic construction;
• Apply guidelines based mostly on sq. dimension and nesting depth to deduce inside fill colours;
• Type recognized squares by dimension earlier than filling, successfully avoiding overwrite conflicts.

Right here’s the generated pseudocode:

1. Let grid be the enter 2D array.
2. Create output_grid as a duplicate of grid.

3. Determine all blue-bordered squares within the grid:
   a. For every attainable top-left nook (i, j):
      i. For every attainable sq. dimension s (s ≥ 3, as much as min(grid top, grid width)):
         - Examine if the sq. of dimension s beginning at (i, j) is totally inside bounds.
         - Examine if all *border* cells of this sq. are Blue (worth = 2).
         - Examine that the *inside* cells (not on the border) don't comprise any Blue (2).
         - If all circumstances are met, file the sq. as (i, j, s).

4. Type the listing of detected blue-bordered squares by dimension in ascending order (smallest first).

5. For every detected sq. (i, j, s), in sorted order:
   a. Decide the fill coloration:
      - If the sq. is the smallest (no different blue-bordered sq. is totally inside it), set fill coloration = Black (8).
      - If the sq. is the most important (no different blue-bordered sq. totally incorporates it), fill coloration = 
        - If there are precisely 2 blue-bordered squares, set fill coloration = Inexperienced (4).
        - If there are three blue-bordered squares within the grid, fill coloration = Yellow (3).
      - If the sq. is nested (not smallest or largest), fill coloration = Black (8).
      - (Extra advanced guidelines could generalize past these based mostly on demonstrations.)

   b. Fill the inside of the sq.:
      For every cell (x, y) strictly contained in the sq. (i+1 ≤ x < i+s−1) and (j+1 ≤ y < j+s−1):
         - If output_grid[x][y] isn't Blue (2), set it to the chosen fill coloration.

6. Return output_grid.

Particular notes:
  - By no means overwrite Blue (2) border pixels. 
  - When filling, later (bigger) squares overwrite earlier (smaller) fills in overlapping areas. 
  - Solely course of legitimate blue-bordered squares (minimal dimension 3x3, full border). 
  - If there are a number of disjoint blue-bordered squares, deal with every independently for fill coloration task as above matching the demonstration logic.

The mannequin clearly reveals structured reasoning. It develops an inside illustration of nested geometric kinds and makes an attempt to use rule-based transformations derived from examples.

Nevertheless, regardless of these advances, the mannequin nonetheless fails to generalize reliably. In novel configurations, it mis-assigns fill colours, falling again on brittle heuristics like size-based precedence or inflexible nesting assumptions. As an example, it would assume the most important sq. is all the time stuffed with yellow, even when that logic not holds in a brand new context. This failure reveals a deeper limitation: even with picture enter, the mannequin doesn’t “see” within the human sense. It doesn’t construct a holistic perceptual illustration of spatial relationships. As an alternative, it converts the picture into symbolic patterns and applies deterministic procedures like flood-fill, sorting, or positional indexing.

In apply, this implies the mannequin causes from inside abstractions, not perceptual grounding. It infers that “smaller squares get black,” or “fill based mostly on dimension rank,” with out totally understanding why these assignments occurred within the demonstrations. Consequently, any deviation from the anticipated format could cause it to misfire.

This implies that whereas multimodal prompting extends the expressive vary of the mannequin, it doesn’t but present the sort of versatile, generalizable visible reasoning that people show. These duties could finally require stronger types of program induction, meta-learning, or hybrid programs that combine perceptual grouping with realized guidelines.

Conclusions

On this examine, I explored whether or not giant language fashions — each text-based and multimodal — can carry out meta-learning from examples on summary reasoning duties. Particularly, I centered on a category of issues from the ARC dataset, the place options require figuring out visible patterns, studying transformations, and generalizing them to novel take a look at inputs.

By means of direct prompting experiments, I discovered that:

• Textual content-based fashions (e.g., o3-mini) usually hallucinate believable algorithms which might be topologically or procedurally sound however solely disconnected from the duty’s visible logic. These fashions depend on symbolic reasoning over tokenized grids, and default to acquainted heuristics like flood-fill, ring propagation, or rule-based updates, whatever the examples supplied.
• Multimodal fashions (e.g., GPT-4 with imaginative and prescient) confirmed a clear enchancment in sample detection and relational reasoning. They efficiently recognized blue-bordered areas and tailored behaviors based mostly on relative dimension or nesting. Nevertheless, their generalization remained fragile: they nonetheless utilized brittle guidelines, equivalent to fastened size-based assignments, and failed in novel layouts that deviated from the demonstrations.

These findings recommend that, even with visible enter, present LLMs don’t “see” as people do. They course of pictures symbolically, not perceptually. Their reasoning is pushed by internally constructed guidelines, not a versatile, visible understanding of shapes, hierarchy, or affordance.

The restrictions I noticed reinforce a central rigidity: few-shot prompting alone, even with pictures, isn’t enough for sturdy abstraction. True generalization seemingly requires:

• Program induction: inferring reusable, structured transformations from examples; 
• Perceptual grounding: growing architectures that parse and manipulate visible scenes compositionally;
• Meta-learning architectures: constructing fashions that adapt their reasoning methods dynamically slightly than making use of pre-learned heuristics;

Right now’s LLMs are astonishing of their breadth, however they’re nonetheless guessing based mostly on priors, not studying to be taught within the human sense. They lack a robust inductive bias for abstraction and transformation. ARC-style duties expose this hole clearly: success requires greater than sample recognition, it requires reasoning from examples in a structured, compositional means. These outcomes should not discouraging, slightly clarifying. We now know the place the ceiling is. And the following era of fashions, these with hybrid architectures, persistent reminiscence, and express meta-learning capabilities may lastly break via it.