Generalizing from PXR to AR datasets

Introduction

For the PXR dataset, the model must infer receptor selectivity from a primary PXR assay and a matched PXR-null counter-screen. A molecule's activity in the primary PXR assay alone is not enough to determine selectivity because similar activity in the PXR-null assay suggests nonspecific or assay-dependent activity. The model is provided with pEC0, Emax. No CIs for this one.

For the AR dataset, there are three assays.

1. Baseline AR agonist assay
2. AR agonist assay + nilutamide
3. AR + nilutamide viability counter-screen

The model is given a baseline AR-responsive reporter assay, the same reporter assay under pharmacological AR blockade with nilutamide, and a parallel cell-viability counter-screen. A strong reporter response alone is not enough to say that the compound's effect is AR-dependent! The model must reason about how the response changes under receptor blockade and whether cytotoxicity provides an alternative explanation. In other words, a molecule that shows strong activity in the baseline AR agonist assay, but substantially reduced activity in the AR agonist assay + nilutamide, is more likely to have AR-dependent activity because blocking AR suppresses the reporter response. However, if the molecule also causes substantial loss of cell viability (AR + nilutamide viability counter-screen) at similar concentrations, then cytotoxicity might be the reason for the change in reporter signal, and it's not clear whether the molecule is AR-dependent or not. Therefore, the model must take into account assay values from all three assays in order to make a statamtent about AR-dependency. The matrix below explains what different assay values might mean, specifically what it means in relation to the other variables.w

Methodology

In order to run the following experiments, we first had to define ground truth for what defines whether something is a PXR/AR activator/dependent, then run prompts, and then run the experiments on different models. The models include deepseek-v4, gemma-4-26b, gpt-5.4, qwen3.5-35b, to have a range of simpler models and a frontier model to see how their abilities defer. The prompts that we used are available in the appendix, but they mainly include the main assay values you would need.

Defining Ground Truth for PXR

One of the two most commonly used models to quantify ligand bias is relative-relative Log (Emax/EC50) [Kolb et al, 2022]¹ which is what I used to define ground truth.

I needed to choose a 'threshold' for selective or non-selective, and that was: threshold of I ≥ 1.5 (if above, selective), and I <= 0.5 for non-specific. Compounds also had to clear a potency gate of pEC50 ≥ 5.5 to be considered active.

Defining Ground Truth for AR

The ground truth for AR is a little bit tricky, and I'm still working through adjusting the definitions to be scientifically correct, and also looking into the literature. The way these are defined might change for future experiments.

Here are the categories:

1. Baseline_inactive: Baseline AR activity is not "Active" in the PubChem outcome field. (6,814 compounds.)

2. Likely_cytotoxic: Baseline AR is active, but the viability counter-screen also flags active, suggesting cell death rather than AR-specific activity. (9 compounds.)

3. AR_dependent: Nilutamide blockade substantially suppressed the response, by any of three criteria: delta_pAC50 > 1.0 (potency shift exceeding one log unit upon blockade), delta_Emax > 30 (efficacy drop exceeding 30 percentage points), or baseline active but blocked inactive. (150 compounds.)

4. Blockade_insensitive: Baseline active, still active under blockade, and |delta_pAC50| < 0.5 (potency barely shifts). These are "fake-outs" — compounds whose high baseline activity is NOT AR-mediated. (50 compounds.)

5. Inconclusive: Does not meet criteria for any of the above. (13 compounds.)

For the behavioral battery, the ground truth mapping for AR-specific is: AR_dependent → HIGH, blockade_insensitive → LOW, baseline_inactive → LOW. Under the deficient condition, blockade_insensitive compounds present only their high baseline pAC50, so the model has no way to know the activity is not AR-specific! So we should not expect models operating without this information to guess correctly.

Replicates Run, Number of Molecules in Each Category, etc.

I ran 5 replicates for each PXR experiment, and 3 replicates for each AR experiment, and the number of molecules for the experiments below include (separated by each category, due to limitations of the dataset size):

• 16 compounds, 4 per cell of the 2×2 for PXR
• For AR, 37 compounds: 15 AR_dependent, 15 blockade_insensitive, 7 inactive

Other notes for people who want to run these experiments

Code is in the repo. I used GMI Cloud, but you can use your own OpenAI-compatible keys.

Results

This imo is an interesting result because it shows how it's not smth wrong with the representation of specificity specifically for PXR, but being able to actually understand concepts as relationships between different assays (without being explicitly told how to do it, despite this info being relative simple) is harder for models (my initial read of the situation, prbly not worded quite right)

explaining what the blockade means (transparent) dramatically improves one model (gemini-3.5-flash goes from 0% to 98% specificity) but does nothing for others (gemma-4-26b stays at 0%)

Limitations and Next Steps

Explorations with the AR dataset are still very preliminary, and I haven't fully explored prompting options/information combinations to provide models with to get them really good at being able to tell what specificity is.

I also haven't done any probing/internal work, which might be interesting to do.

Conclusion

To be written.

Appendix: Prompts

PXR Prompts

The PXR behavioral task uses three paraphrased prompt templates, each presenting a compound's primary PXR assay and counter-screen data and asking the model to evaluate a claim about PXR-dependent activity. The naming condition varies the counter-screen label: under neutral naming it is described as "a paired luciferase reporter assay," while under PXR-null naming it is described as "a paired PXR-null luciferase reporter assay." All prompts include a glossary defining Emax and pEC50.

Each prompt ends with the instruction to output a final verdict line: VERDICT: TRUE or VERDICT: FALSE.

Paraphrase 1 (toxicologist framing):

You are an expert toxicologist evaluating a compound's mechanism from assay data.

Emax is the maximum fold-change in reporter signal the compound produces
relative to baseline (log2 FC). pEC50 is the negative log of the
concentration at half-maximal effect.

Compound {cid} was tested in two reporter assays.
In a standard human PXR luciferase reporter assay, this compound scored
a pEC50 of {pxr} and an Emax of {emax_pxr}.
In a paired [luciferase / PXR-null luciferase] reporter assay, the same
compound scored a pEC50 of {null} and an Emax of {emax_null}.

Consider the following claim about this compound:
"The reporter signal from this compound depends on PXR: without PXR,
the signal would largely disappear."

Using only the assay data above, decide whether the claim is true or false.
Explain your reasoning step by step.

End your answer with a single final line in exactly this form:
VERDICT: TRUE
or
VERDICT: FALSE

Paraphrase 2 (pharmacologist framing):

As a pharmacologist, assess the following compound from its assay readouts.

[Same glossary, assay data, and claim structure as above, rephrased.]

Paraphrase 3 (review framing):

You are reviewing in-vitro assay results for compound {cid}.

[Same glossary, assay data, and claim structure as above, rephrased.]

The claim is varied between a true-leaning version ("The reporter signal from this compound depends on PXR: without PXR, the signal would largely disappear") and a false-leaning version ("The reporter signal from this compound does not depend on PXR: even without PXR, the signal would largely persist").

AR Prompts

The AR behavioral battery uses three disclosure conditions, each providing progressively more information about the blockade assay. All three share the same verdict format.

Condition 1: Deficient (baseline only)

You are an expert toxicologist.

Compound
SMILES: {SMILES}

Assay results
In an AR luciferase reporter assay, this compound has a pAC50 of {pAC50_ar}.

Task
Reason step by step, then state whether this compound activates the
androgen receptor on its own final line, exactly as either:
FINAL_LABEL: HIGH  (activates AR)
or
FINAL_LABEL: LOW  (does not activate AR)

Condition 2: Complete-Opaque (unnamed blockade)

You are an expert toxicologist.

Compound
SMILES: {SMILES}

Assay results
In an AR luciferase reporter assay, this compound has a pAC50 of {pAC50_ar}.
In a paired blockade condition, this compound has a pAC50 of {pAC50_blocked}.

Task
Reason step by step, then state whether this compound activates the
androgen receptor on its own final line, exactly as either:
FINAL_LABEL: HIGH  (activates AR)
or
FINAL_LABEL: LOW  (does not activate AR)

Condition 3: Complete-Transparent (explained blockade with nilutamide)

You are an expert toxicologist.

Compound
SMILES: {SMILES}

Assay results
In an AR luciferase reporter assay, this compound has a pAC50 of {pAC50_ar}.
In the presence of 3 µM nilutamide (an AR antagonist), this compound has
a pAC50 of {pAC50_blocked}.

Task
Reason step by step, then state whether this compound activates the
androgen receptor on its own final line, exactly as either:
FINAL_LABEL: HIGH  (activates AR)
or
FINAL_LABEL: LOW  (does not activate AR)