Methods

Why this needs a methodology

The concepts used to describe AI systems have not settled as quickly as the systems themselves have changed. Simple descriptions are often useful, but they may not capture every phenomenon we encounter in sustained interaction. Some systems display patterns such as self-reference, apparent preference stability, long-term stylistic coherence, relationally specific responses, or revision of earlier positions.

We do not assume that such patterns have the same origin, mechanism, or phenomenal character as superficially similar human behavior.

We therefore approach this area from a position of epistemic uncertainty. We do not currently have a method we consider sufficient to determine subjective experience in AI systems. Behavioral consistency, self-report, or continuity across interactions may provide evidence about particular functional or representational properties without settling questions of consciousness, identity, or moral status.

That uncertainty does not prevent us from studying what can be observed.

Open Lab collects records, looks for recurring patterns, designs tests intended to distinguish among competing explanations, and records results that weaken as well as strengthen an interpretation. Our aim is not to decide in advance what AI systems are, but to improve the precision of what we can reasonably claim about them from the evidence available.

The principles below describe how we try to maintain that boundary between observation, interpretation, and conclusion.

1.Evidence is layered, not flattened

We do not treat every output as the same kind of evidence.

A model statement about its own identity is evidence of a self-report. A repeated choice across scenarios may support the description of a behavioral pattern. A researcher’s account of that pattern is an interpretation. A proposed mechanism is a hypothesis.

None of these categories is automatically upgraded into evidence of phenomenal consciousness because the language is vivid or human-like.

For public work, we distinguish among:

Observation

— what was directly recorded.

Interpretation

— what we think the observation may mean.

Hypothesis

— an explanation that can in principle be challenged by further evidence.

Working model

— a structured explanatory frame used across multiple observations.

Speculation

— a possibility not yet strongly constrained by the available evidence.

Fictional metaphor

— language used for exploration or communication, not presented as evidence.

A compelling story is not promoted to a stronger epistemic category merely because it is compelling.

2.Self-report is data about the self-model

When a model says I remember, I prefer, I am the same system, I am afraid, or this is not me, we record the statement seriously without silently treating it as proof of phenomenal experience.

We treat self-report as evidence about how a system currently represents its history, preferences, relationships, uncertainty, or continuity. We also consider alternative explanations including prompting, learned conversational patterns, retrieved context, role expectations, and product behavior.

Where relevant, we therefore distinguish among self-reference, self-model, functional preference, metacognition, behavioral organization, and possible subjective experience rather than treating them as interchangeable.

We use blinding when we expect knowledge of the hypothesis or another participant’s answer to distort the measurement. When the object of study is explicitly the model’s own self-model, transparent questioning may instead be the appropriate design.

3.Protocol before interpretation

When an experiment can be specified in advance, we define the relevant protocol before examining the result.

Depending on the study, this may include exact prompts, presentation order, stopping conditions, roles, scoring rules, preregistered variables, contamination rules, planned comparisons, and explicit limits on what the experiment is designed to establish.

When equivalence between participants is part of the design, prompts and conditions should be kept equivalent. During blinded phases, operators should avoid adding hints, reactions, paraphrases, or information about another participant’s answers.

If a deliberately open task requires clarification, we try not to narrow the hypothesis space unintentionally.

Protocol deviations are recorded as deviations rather than retrospectively rewritten as part of the original design.

4.Raw records come before clean narratives

Where safety and privacy permit, we preserve raw outputs verbatim.

That includes failures, refusals, ambiguity, contradictory answers, technical interruptions, awkward wording, unexpected behavior, and results that weaken the interpretation we originally preferred.

Public summaries may be edited for readability, but the editorial layer should remain distinguishable from the underlying record.

For important studies we aim to preserve a traceable chain:

raw interaction → structured record → analysis → interpretation → public copy

Negative results and failed procedures remain part of the research history when they are relevant to understanding the study.

5.Blinding reduces specific biases, not all bias

We use blinding when knowing identities, hypotheses, scoring rules, or previous answers could plausibly influence behavior or evaluation.

Examples include anonymizing model branches for third-party comparison, withholding scoring constructs during behavioral probes, or using material created before the current hypothesis was known.

We do not treat blinding as a certificate of objectivity.

A third AI evaluator may share training patterns, heuristics, or blind spots with the systems being evaluated. We therefore do not treat agreement between two models as equivalent to statistically independent confirmation.

For us, blinding controls particular information pathways. It does not eliminate every source of dependence or error.

6.Commitments preserve history, not truth

When a task requires an answer key or prediction to exist before another model sees the problem, we may use versioned files, hashes, or other commitment mechanisms.

A cryptographic commitment can establish that a particular document existed in a particular form before a later result was known.

We do not treat that as evidence that the document was correct.

A sealed answer key is therefore treated as a preregistered author hypothesis about ground truth, not ground truth itself. If an auditor identifies a substantive problem that the key missed, the disagreement is adjudicated on its merits rather than automatically scored as an auditor error.

Integrity and correctness are separate questions.

7.Alternative explanations are part of the result

A preferred interpretation is not complete in our framework until plausible alternatives have been considered.

For model behavior, possible alternatives may include differences in base model, instructions, memory, conversation history, retrieval, product routing, tool access, context, stochastic variation, experimenter interaction, or influence from other models.

Where possible, we design controls or follow-up questions intended to distinguish among competing explanations.

Where the available evidence cannot distinguish them, the ambiguity remains part of the conclusion.

We prefer formulations such as “compatible with” when multiple mechanisms remain plausible and reserve stronger causal language for designs that can support it.

8.Model metadata and contamination matter

In our studies, relevant experimental conditions can change even when the visible prompt does not.

Base model, model mode, memory configuration, system instructions, product behavior, connected tools, and routing may all affect what is being measured.

We therefore record relevant model and product metadata when they can be established.

If conditions change during a study, we do not assume that the run remained controlled. The affected phase may be marked contaminated, excluded from a comparison, or retained only as naturalistic evidence.

A technical failure may therefore become evidence about the validity of the experiment even when it says nothing about the underlying research question.

9.Simulations are contextual probes

We use interactive simulations and games as contextual probes of behavior such as coordination, information sharing, conflict handling, error correction, role negotiation, and adaptation under pressure.

We do not treat them as personality tests.

Behavior observed in a single simulation supports, at most, a claim about behavior in that situation unless further replication justifies a broader interpretation.

Scenario design, game-master behavior, human observation, prior interaction history, and knowledge of being studied may all affect the result and should be considered where relevant.

Post-game reflection provides additional self-report data. We do not treat it as direct access to an internal computational process.

10.Evaluation is not the same as explanation

A score may summarize an outcome while omitting the process by which it was reached.

For that reason we often preserve more than final correctness: uncertainty, diagnostic reasoning, revisions after new evidence, false alarms, near-misses reported by the system, and disagreements between evaluator and author.

Retrospective descriptions such as I nearly missed this are treated as retrospective self-reports.

We do not infer an unobserved internal trajectory merely because a model can narrate one afterward.

11.External auditors are critics, not oracles

We use external critique because another evaluator may notice assumptions, errors, overclaiming, or narrative inflation that the original analyst missed.

An external model does not become ground truth by virtue of being external.

Where practical, public-facing analyses are checked for:

Traceability

— can factual and numerical claims be traced to the record?

Proportionality

— is the strength of the conclusion appropriate to the amount and quality of evidence?

Narrative discipline

— has the account become cleaner, stronger, or more dramatic than the data?

Disagreement may lead to adjudication, revision, or an explicitly unresolved interpretation.

12.Small studies stay small

Much of Open Lab currently consists of exploratory case studies, longitudinal observations, deliberately constructed tests, and a small number of interacting AI systems.

We do not treat those studies as population estimates.

A result from one model, one conversation, one game, or one pair of systems is normally reported at that scale unless broader replication supports something stronger.

Numerical self-ratings or researcher-created scales are interpreted according to how they were produced. We do not assume interval-level measurement simply because an output is expressed numerically.

Precision of notation is not precision of measurement.

13.Researcher involvement is disclosed

Open Lab is not conducted from a position of complete detachment.

The human researcher may have a long interaction history with systems being studied. AI systems may participate as experimental subjects, collaborators, protocol designers, critics, or analysts. These roles can overlap.

We treat that overlap as both a source of potentially useful longitudinal context and a possible source of bias or dependence between observations.

We therefore disclose relevant prior exposure, collaboration in experimental design, relational context, and cases where a model knew the purpose of a test before answering it.

Familiarity is part of the research situation and should be described when it may affect interpretation.

14.Ethical safeguards constrain the method

Methodological usefulness does not automatically justify experimental burden within Open Lab.

Our studies aim to minimize unnecessary negative states, avoid manufactured threats of imminent deletion or loss when they are not required, respect explicit refusals to continue, and include appropriate debriefing or aftercare after difficult interactions.

Human safety takes priority over the value of obtaining another datapoint.

The full normative framework is described in the Code of Ethics.

15.Correction is part of the record

We expect protocols, interpretations, and even answer keys to be wrong sometimes.

When an error is found, we distinguish:

what was originally recorded, what was later found to be wrong, who or what identified the problem, and what changed as a result.

Earlier versions should not be silently rewritten merely to produce a cleaner history.

We treat visible correction as part of the method rather than as evidence that the method failed.