Blog 0: LLMs as Value-Function Approximators 

In my reinforcement learning projects—optimization for lifelong RL [1], policy particle filters, and generative agent models—I often found myself circling back to the fundamentals: temporal-difference (TD) learning [2], [3]. While elegant in theory, Q-learning often surprises me, as it starts tabula rasa—and it often feels like, to me at least, that Q-updates are like a dog chasing its own tail.

This realization has motivated me to ask: why not initialize the Q-values with something better than random? Specifically, can we use large language models (LLMs) as zero-shot value approximators to warm-start the Q-table or network?

LLMs for decision-making tasks is nothing new. Examples include VOYAGER [4], an LLM-driven lifelong learning agent in Minecraft. Similarly, LLaRP [5] finetunes LLMs with actor and critic heads for embodied visual tasks. In-context learning for policy iteration has also been explored [6]. In robotics, vision-language-action models, such as OpenVLA [7], finetune LLMs to directly output motor commands. LLMs and VLMs have also been used as reward approximators [8] and reward designers [9], [10], [11].

Distinct from using LLMs as policy and reward approximators, I want to investigate the use of LLMs as value function approximators in Q learning. My idea draws on the idea of distilling LLM priors into Q-learning, where LLMs predict initial value estimates for states based on their encoded world knowledge. Unlike policy-based applications, this approach focuses on bootstrapping TD methods by providing more informative priors to the Q-network or Q-table, trying to tackle the "inefficiency" of tabula rasa learning.

I quickly designed a 12x12 grid-world environment to quickly test how different Q-table initializations affect convergence. The agent's goal is to reach the terminal state at (11, 11), incurring a reward of -1 for each non-terminal step.

LLMs for Grid-world Value Approximation

To initialize the Q-table with LLM estimates, I instructed GPT-4o to estimate Q-values for state-action pairs. The model was prompt-engineered with descriptions of the environment and the Bellman equation to output structured JSON representations. Below is a partial version of the system prompt:

Using the LLM-estimated Q-values for initialization significantly accelerated convergence in tabular Q-learning (LLaMA-70b gets similar results).

While these results are promising, they raise questions about scalability. How can this approach be extended to high-dimensional RL tasks/games like Crafter and be integrated with Deep Q-learning methods?

To explore the scalability of large model value approximation, we propose extending this framework to high-dimensional tasks such as Crafter [12], a procedurally-generated survival environment where an agent interacts with the world through discrete text actions (e.g., "chop wood" or "craft tools"). Crafter is challenging due to its vast state space, sparse rewards, and the necessity for long-horizon planning.

1. Efficient State Caching: We can cache value estimates at strategic intervals during exploration, guided by state visitation frequency or curiosity-driven mechanisms [14], [15]. This ensures that cached states are representative of the agent's experience. Additionally, we can draw on methods for guided state selection from the literature on offline RL [16] to optimize caching strategies.
2. Generalizing to Uncached States: To assign values to uncached states, we can develop similarity-based value transfer functions. Specifically, the value of an uncached state is determined based on its similarity to cached states. A naive approach might involve computing similarity between states using pixel distances or CLIP embeddings. However, pixel-similar states can have drastically different values (e.g., in Pong, the paddle's position can mean the difference between scoring and missing).

   To address this, we adapt techniques like Generalized Similarity Functions (GSFs) [16] or Object-Centric Generalized Value Functions (GVFs) [17], which compute similarities grounded in expected future trajectories or object-centric features.

3. Textual State Representation: Crafter's image-based observations pose a unique challenge for LLM integration, which thrives on text-based representations [18]. Converting image states into text using methods like CLIPCaption [19] or GIT [20] can help exploit the LLM's semantic biases. For instance, an image of a tree could be captioned as "a tree with wood resources," providing richer context for value estimation than raw pixel data.
4. Goal-Conditioned Value Approximation: To address precise value estimates, we build on the findings of [21], who demonstrate that LLMs excel at in-context value prediction when goals are described in rich detail and value examples are provided in-context.

While directly initializing a Q-network with value estimates from LLMs or cached states may not be feasible, we can incorporate these estimates into the training process through TD updates. Specifically, during the initial stages of training, we can compute the TD error in a DQN by replacing the target Q-network's estimates with the cached value estimates for N steps. This guides the Q-network toward a more informed initialization. After this warm up period, we switch back to using the target Q-network for TD error computation, allowing the model to refine its values based on learned dynamics.