AI News
All stories
Official

OpenAI's LifeVal benchmark tests assistants on personal tasks built from real chats

OpenAI introduces LifeVal, a set of 169 agent tasks based on 148 life tasks ranked by 3,837 US consumers, where the most common failure is a wrong answer or recommendation.

Research · · 1 min read

OpenAI has released LifeVal, a benchmark of 169 agent tasks for measuring how well AI assistants handle personal life admin. For every model tested, the most common failure is a wrong answer or recommendation. Often the model missed a fact in the user's own past chats.

The benchmark is built from a survey of 3,837 US consumers. They ranked 148 consumer task categories, such as managing finances, moving apartments, searching for a job and finding childcare, by what they would pay an assistant to handle them. The paper, first posted to arXiv on November 4, 2026, says these categories together represent over 1.7 trillion US dollars in annual willingness-to-pay. The authors cross-checked the survey against the American Time Use Survey and the Multinational Time Use Study.

Task creators had to pass an expertise threshold. That included personally completing the task several times and following the results through in their real lives for over a year. Each creator then wrote a personalized task using real ChatGPT conversations and memories, reference files and personal context, along with a rubric describing the best possible outcome. Every task passed at least three rounds of human quality control. Per the DAIR.AI Academy summary of the paper, each task also ships with a starting state, a reinforcement learning environment and an offline web. A second loop has one model attempt a task while another classifies failures as rubric, prompt, environment or model problems.

Results were reported across GPT-5.2, 5.4, 5.5, 5.6 and GPT-6 Astra, with rewards rising across releases. On rubric criteria tied to a single source, the paper reports win rates of 33.3% for past conversations, 47.8% for saved memories and 55.8% for task context. Past conversations are therefore the weakest of the three sources. The 33% figure in the announcement is the rounded version of that number.

Why it matters

Personal assistants depend on recalling past conversations, and the reported win rate is lower there than for saved memories, which affects rollout decisions for memory-based products.

What you can do

Test assistant recall of past conversations separately from saved memory before rolling out a personal assistant.

Original source

Where we saw it

1 source

DAIR.AI (Elvis Saravia)

AI wrote this summary from the sources listed, and nobody reviewed it individually before publication. Check the original source before you rely on it. How it works