Dataset
First 50 rows of validation
| datasetstring | agent_ref | responses_create_params | expected_answerfloat64 | answer_typestring | output_regexstring | propertystring | methodstring | question_licensestring | metadata | question_familystring | matchJson |
|---|---|---|---|---|---|---|---|---|---|---|---|
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Determine the number of heavy atoms in this chemical structure. Provide a whole number.\n\nCCC(O)c1ccc(O)cc1\n\nPlace your final answer within double angle brackets at the end of your response: <<integer>>. The final result should be a single i… | 11 | float | <<\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | HeavyAtomCount | direct | CC-BY-SA-4.0 | {"uuid":"99143c3d4b1aa772c2d2da48ec029fff","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL181890"}]} | tier1_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Determine the Fsp3 of the following SMILES. Give a numeric value.\n\nCN1CCN(c2ccc([N+](=O)[O-])cc2NC(=O)c2ccc(C(F)(F)F)cc2Cl)CC1\n\nEnd your response with Final Answer: ||floating_point_number||, where floating_point_number is the final answer.… | 0.31578946 | float | Final Answer:\s*\*{0,2}\|{0,2}\s*([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) |
| direct |
| CC-BY-SA-4.0 |
| {"uuid":"4d275127cace1d0c83d38767c382e666","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL3799267"}]} |
| tier1_rdkit |
| {"abs_tol":0.000003,"rule":"abs_window"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Identify the number of heterocyclic rings (containing at least one heteroatom) in this molecule.\n\nCOC(=O)C1=C(C(=O)OC)[C@@H]2Nc3ccccc3[C@@]23C[C@@H](C(=O)OC)N(S(=O)(=O)c2ccc(C)cc2)C3=N1\n\nConclude your response with \"**Final answer is: inte… | 3 | float | \*{0,2}Final answer is:\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | NumHeterocycles | direct | CC-BY-SA-4.0 | {"uuid":"ebd73f242b2a0848e8e33f1e01dd043a","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL1396296"}]} | tier1_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"What is the number of primary amides in this compound?\n\nCCCCCCCCCCCCCC(=O)NCCCC[C@H](NC(=O)[C@H](Cc1c[nH]c2ccccc12)NC(=O)[C@H](Cc1cnc[nH]1)NC(=O)[C@@H]1CCC(=O)N1)C(=O)N[C@@H](Cc1cnc[nH]1)C(=O)N[C@@H](CC(=O)O)C(=O)N[C@@H](Cc1c[nH]c2ccccc12)C(=… | 1 | float | <?Answer:\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_priamide | direct | CC-BY-SA-4.0 | {"uuid":"32659b70472fd5734b05e963254a4966","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL2088510"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"What is the count of stereocenters with unspecified configuration in this compound?\n\nCC(=O)Nc1nnc(N2CCN(C(C)c3ccc4c(c3)OC(C)C4)CC2)s1\n\nPlace your final answer within double angle brackets at the end of your response: <<integer>>. The final … | 2 | float | <<\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | NumUnspecifiedAtomStereoCenters | direct | CC-BY-SA-4.0 | {"uuid":"39df465b2a073505e13fd53316b26809","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL5930443"}]} | tier1_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Does this compound have an amide group (C(=O)N)?\n\nCOC1(OC)COc2cc(O)ccc2-c2cc(O)c(O)cc2C1\n\nPut the correct answer at the end like this: Correct Answer: [0_or_1]. The final result should be 0 or 1. (1 = yes/present, 0 = no/absent)"}]} | 0 | bool | Correct Answer:\s*\*{0,2}\[?\s*([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | HasAmide | direct | CC-BY-SA-4.0 | {"uuid":"06c8d0b3d8bc5a95f69279611eb6d990","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL2207726"}]} | tier2_rdkit |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Count the urea groups in this molecule. Give your answer as an integer.\n\nO=C(O)CCCN1C[C@H](C(=O)O)Oc2c(NC(=O)c3ccc(OCCCCc4ccccc4)cc3)cccc21\n\nPlace your final answer at the end of your response, enclosed within XML-style tags as follows: `<f… | 0 | float | <final_answer>\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_urea | direct | CC-BY-SA-4.0 | {"uuid":"fc4ce8a3ae13dab009b89c0c9c80c766","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL3342963"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Does the following compound satisfy Veber's oral bioavailability criteria (TPSA ≤ 140 Ų, rotatable bonds ≤ 10)?\n\nO=C(Nc1ccc(Oc2ccc(C(=O)c3ncc[nH]3)cc2)cc1)Nc1ccc(Cl)c(C(F)(F)F)c1\n\nPlace the final answer within \\boxed{}. Do not modify the … | 1 | bool | \\boxed\{\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | PassesVeber | direct | CC-BY-SA-4.0 | {"uuid":"805e005e0d4a8382213ed65ec038b776","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL5284834"}]} | tier2_rdkit |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Identify the number of hydrazone groups (C=N-NH) in the following compound. Give your answer as an integer.\n\nO=C(N/N=C(\\CC(=O)c1cccs1)C(F)(F)F)c1ccncc1\n\nAt the end of your response, state the answer in the following format: 'Correct answer… | 1 | float | Correct answer is\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_hdrzone | direct | CC-BY-SA-4.0 | {"uuid":"5c7ec8c11966309cbe1f12bded4eadc3","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL3199529"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"How many total quaternary nitrogens in the molecule below? Give your answer as an integer.\n\nCOc1cc2c(cc1OC)[C@H](c1cc(OC)c(OC)c(OC)c1)[N@@+](C)(CCCOC(=O)/C=C(\\Cl)C(=O)OCCC[N@+]1(C)CCc3cc(OC)c(OC)cc3[C@H]1Cc1cc(OC)c(OC)c(OC)c1)CC2\n\nEnd your… | 2 | float | Final Answer:\s*\*{0,2}\|{0,2}\s*([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_quatN | direct | CC-BY-SA-4.0 | {"uuid":"00e9827ddf332eaf9b143f9eb38a5442","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL1182997"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"What is the predicted quantitative estimate of drug-likeness (QED) for this compound? Give a numeric value.\n\nCCN(CC)CCn1c2ccc(OC)cc2c2sc3cc(OC)ccc3c(=O)c21\n\nAt the end of your response, state the answer in the following format: 'Correct ans… | 0.4418505 | float | Correct answer is\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | qed | direct | CC-BY-SA-4.0 | {"uuid":"de7fcaf87419956e98bb62042dba42ef","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL558498"}]} | tier2_rdkit | {"abs_tol":0.02,"rule":"abs_window"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Identify the number of oxazole rings, includes bicyclics (c1ocnc1) in this compound.\n\nCOC(=O)c1cc2occc2n1Cc1nc(-c2ccc(Cl)cc2)oc1C\n\nPlace the final answer within square brackets at the end of your response: Answer is [integer]. The final res… | 1 | float | Answer is\s*\*{0,2}\[?\s*([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_oxazole | direct | CC-BY-SA-4.0 | {"uuid":"3cd93d83d68aeaaabff79e675de6a8aa","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL1584279"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"What is the count of morpholine rings (O1CCNCC1) in this chemical structure? Provide a whole number.\n\nCN(C)c1ccc(C(=O)N2CCCC(n3cncn3)C2)nn1\n\nProvide your final answer as '<Answer: integer>', where integer is the answer. The final result sho… | 0 | float | <?Answer:\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_morpholine | direct | CC-BY-SA-4.0 | {"uuid":"6a077aa610de0d4c5a44889d8e3c3e8d","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL3492129"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"What is the number of hydroxylamine groups in this molecule?\n\nCCCCCc1nc2cc(/C=C/C(=O)NO)ccc2n1CCN(C)CC\n\nEnd your response with the final answer on a new line in the format: Response: integer, where integer is the answer. The final result sh… | 1 | float | Response:\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_N_O | direct | CC-BY-SA-4.0 | {"uuid":"646bddd82ef001a6b422a97bd20ae4c7","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL1830419"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"How many total aryl methyl sites for hydroxylation in the molecule below?\n\nCOc1cccc(OC)c1-c1ccc(C[C@H](NC(=O)[C@@H]2CCCN2c2cnccn2)C(=O)O)cc1\n\nPlace your final answer within double angle brackets at the end of your response: <<integer>>. The… | 0 | float | <<\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_aryl_methyl | direct | CC-BY-SA-4.0 | {"uuid":"065cb26199b3a6017a455c17e1393726","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL70107"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"How many total saturated heterocyclic rings in the molecule below? Provide a whole number.\n\nNc1n[nH]c2ccc(C(=O)N3CC[C@@H](c4ccc(Cl)cc4)C3)cc12\n\nProvide your final answer as '<Answer: integer>', where integer is the answer. The final result … | 1 | float | <?Answer:\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | NumSaturatedHeterocycles | direct | CC-BY-SA-4.0 | {"uuid":"1cad4c57a01dbc2be5291f03f901ea43","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL3955876"}]} | tier1_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"How many total nitrogen atoms with exactly one hydrogen (NH), includes amide NH, sulfonamide NH, aromatic NH (pyrrole, indole), and secondary aliphatic amines ([NH1,nH1]) in this compound? Provide a whole number.\n\nCc1cc(Cl)cc2c1C[C@H](N(C)C)[… | 6 | float | Final Answer:\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_NH1 | direct | CC-BY-SA-4.0 | {"uuid":"8b4f9279c0c433208cee39e1eb9f3d68","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL5875359"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"How many sulfones in the following SMILES?\n\nNS(=O)(=O)CCNC(=O)C(c1nc2ccc(-c3ccc(F)nc3)cc2s1)S(=O)(=O)CCN1CCOCC1\n\nYour final answer must be enclosed in double parentheses. Do not modify the required answer format with any markdown, code fenc… | 1 | float | \(\(\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_sulfone | direct | CC-BY-SA-4.0 | {"uuid":"d33b8f99734ff05ceb0e8f3306a9c48d","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL3677638"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Identify the number of nitro benzene ring substituents (Ar-NO2) in this compound.\n\nC[C@H](C[C@H](OC(=O)c1cccc([N+](=O)[O-])c1)[C@@H](OC(=O)c1cccc([N+](=O)[O-])c1)C(C)(C)OC(=O)c1cccc([N+](=O)[O-])c1)C1=C2C[C@H](OC(=O)c3cccc([N+](=O)[O-])c3)[C@… | 4 | float | \*{0,2}Final answer is:\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_nitro_arom | direct | CC-BY-SA-4.0 | {"uuid":"bd4f60be623fee5bb4832c07845265f5","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL574129"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Count the sulfonamides in this molecule. Give your answer as an integer.\n\nCCO[C@H](C(=O)Nc1nnc(N[C@@H]2CCN(c3nccnn3)C2)s1)c1ccccc1\n\nProvide your final answer as '<Answer: integer>', where integer is the answer. The final result should be a … | 0 | float | <?Answer:\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_sulfonamd | direct | CC-BY-SA-4.0 | {"uuid":"258ad7a75dceb0784f823984a0041e36","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL5905353"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Identify the number of aromatic rings in this compound.\n\nCNc1cc(Nc2cccn(C3CCOCC3)c2=O)nc2c(C(=O)NC3CC[C@]3(O)C(F)(F)F)cnn12\n\nConclude with (Answer: integer), where integer is the final answer. The final result should be a single integer."}]… | 3 | float | \(?Answer:\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | NumAromaticRings | direct | CC-BY-SA-4.0 | {"uuid":"4a79c02eb296b0f65c251560aa153002","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL5839492"}]} | tier1_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Identify the number of barbiturate groups in the following SMILES.\n\nCOc1ccc(-c2ccc(CNC(=O)c3nc(=O)[nH][nH]3)s2)cc1\n\nYour final answer must be enclosed in double curly braces. Do not modify the required answer format with any markdown, code … | 0 | float | \{\{\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_barbitur | direct | CC-BY-SA-4.0 | {"uuid":"455b948e4d9237fb1bdff0bebc06ec70","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL4928651"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"How many total rotatable bonds (acyclic single bonds; excludes bonds to terminal CH₃, amide C(=O)–N, and sulfonamide S–N) in this compound? Give your answer as an integer.\n\nCc1c2c(cn1C)CCc1cc(S(=O)(=O)c3ccccc3)c(=O)[nH]c1-2\n\nEnd your respon… | 2 | float | Final Answer:\s*\*{0,2}\|{0,2}\s*([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | NumRotatableBonds | direct | CC-BY-SA-4.0 | {"uuid":"588689592b9c687b4bc84d87965ec131","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL2181063"}]} | tier1_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"How many total isocyanates in the following SMILES?\n\nC=C(C)[C@@H]1CC[C@]2(N=C=O)CC[C@]3(C)[C@H](CC[C@@H]4[C@@]5(C)CC[C@H](O)C(C)(C)[C@@H]5CC[C@]43C)[C@@H]12\n\nConclude your response with \"Final Answer = integer\", where integer is the answe… | 1 | float | Final Answer\s*=\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_isocyan | direct | CC-BY-SA-4.0 | {"uuid":"503c5f310741cade6653b90a2067745c","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL566478"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"How many saturated all-carbon rings in this compound?\n\nN=C(NC1CCCCC1)c1ccccc1-c1ccccc1\n\nProvide your final answer at the end of the response as: Answer: *integer*, where integer is the answer. The final result should be a single integer."}]… | 1 | float | Answer:\s*\*{0,2}\*?\s*([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | NumSaturatedCarbocycles | direct | CC-BY-SA-4.0 | {"uuid":"9a41c99dbdb23bf2a648ef67e663fc56","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL1860146"}]} | tier1_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Count the pyridine rings in this compound. Provide a whole number.\n\nNS(=O)(=O)c1ccc(N/C=C/C(=O)c2ccc(Cl)s2)cc1\n\nPlace your final answer at the end of your response, enclosed within XML-style tags as follows: `<final_answer>your answer here<… | 0 | float | <final_answer>\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_pyridine | direct | CC-BY-SA-4.0 | {"uuid":"d8dd8d5fbe4007c31861ebfae6ec9d0b","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL579560"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Identify the number of aromatic all-carbon rings in this compound.\n\nCCOC(=O)c1cnc(NC2CCN(Cc3ccc(F)cc3F)CC2)nc1Oc1c(C)cc(-c2ccc(C#N)cc2)cc1C\n\nPlace the final answer within square brackets at the end of your response: Answer is [integer]. The… | 3 | float | Answer is\s*\*{0,2}\[?\s*([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | NumAromaticCarbocycles | direct | CC-BY-SA-4.0 | {"uuid":"38c45081e8cef787964ef9b8fd328788","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL5613851"}]} | tier1_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"How many imidazole rings in the following SMILES?\n\nCc1cn(CC(=O)N(CCNC(=O)CN(CCNC(=O)CN(CCNC(=O)CN(CCNC(=O)CN(CCNC(=O)CN(CCNC(=O)CN(CCNC(=O)CN(CCNC(=O)CN(CCNC(=O)CN(CCNC(=O)CN(CCNC(=O)CN(CCNC(=O)CN(CCNC(=O)CN(CCNC(=O)CN(CCNC(=O)CN(CCNC(=O)CN(C… | 12 | float | <<\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_imidazole | direct | CC-BY-SA-4.0 | {"uuid":"5df14492b14dbfd0d3738515a946f667","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL1253240"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"How many total oxime groups (C=N-OH) in the following SMILES?\n\nO=C(Nc1ccnn1C1CCN(Cc2cccc3nonc23)CC1)c1cccc(F)c1\n\nPlace the final answer within square brackets at the end of your response: Answer is [integer]. The final result should be a si… | 0 | float | Answer is\s*\*{0,2}\[?\s*([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_oxime | direct | CC-BY-SA-4.0 | {"uuid":"d36dec831c7ea5e51862d238c8194a7c","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL4935310"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Determine the number of beta lactams (N1C(=O)CC1) in the molecule below.\n\nC=CCNC(=O)[C@H]1O[C@@H](n2cnc3c(N)ncnc32)[C@H]2OC(C)(C)OC12\n\nYour final answer must be enclosed in double curly braces. Do not modify the required answer format with … | 0 | float | \{\{\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_lactam | direct | CC-BY-SA-4.0 | {"uuid":"ada806d398ba30b6ad06d20ce48583ab","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL321203"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"How many dihydropyridines in the following compound?\n\nCCOC(=O)C1=C(C)NC(C)=C(C(=O)OCC)C1CCc1ccccc1\n\nPlace the final answer within square brackets at the end of your response: Answer is [integer]. The final result should be a single integer.… | 1 | float | Answer is\s*\*{0,2}\[?\s*([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_dihydropyridine | direct | CC-BY-SA-4.0 | {"uuid":"6a4253aaf39416759c054429a4f7cd41","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL3409642"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Identify the number of cyclic esters (lactones) in the following compound. Give your answer as an integer.\n\nCC[C@@]1(C(=O)NCCN2CCCC2)OC(=O)c2c1cc1n(c2=O)Cc2cc3ccccc3nc2-1\n\nPlace the final answer within square brackets at the end of your res… | 1 | float | Answer is\s*\*{0,2}\[?\s*([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_lactone | direct | CC-BY-SA-4.0 | {"uuid":"a54def7473042729d3ef2ffb44c3e12e","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL511999"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"What is the number of thiazole rings, including bicyclics in this compound? Provide a whole number.\n\nCc1csc(Nc2nccc(-c3sc(C4CCNCC4)nc3-c3ccc(F)cc3)n2)n1\n\nConclude your response with: Correct Answer >> integer. The final result should be a s… | 2 | float | Correct Answer >>\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_thiazole | direct | CC-BY-SA-4.0 | {"uuid":"2e7b49b48e99e0710f7ada81a87560f3","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL4593717"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"How many nitroso groups, excluding NO2 in this compound?\n\nCCc1cc(C(=O)N2CCCCC2)c2c(C)nn(-c3ccc(OC)cc3)c2n1\n\nAt the end of your response, state the answer in the format: Final Answer: integer. The final result should be a single integer."}]} | 0 | float | Final Answer:\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_nitroso | direct | CC-BY-SA-4.0 | {"uuid":"8acaa67ba449d9b1324462ab4bcf56b8","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL5027654"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"How many aliphatic heterocyclic rings in the following compound?\n\nCCC1=C(O)c2c(oc(-c3ccc4c(c3)c3ccccc3n4CC)cc2=O)C(CC)(CC)C1=O\n\nProvide your final answer as '<Answer: integer>', where integer is the answer. The final result should be a sing… | 0 | float | <?Answer:\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | NumAliphaticHeterocycles | direct | CC-BY-SA-4.0 | {"uuid":"0311a472111e9e4e8d556a9855a36526","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL1689503"}]} | tier1_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"What is the TPSA (in Ų) of the following SMILES? Give a numeric value.\n\nC[C@@H](Cn1ccc2ccc3ncccc3c21)NCc1ccc2ccccc2c1\n\nProvide your final answer as '<Answer: floating_point_number>', where floating_point_number is the answer. The final res… | 29.85 | float | <?Answer:\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | TPSA | direct | CC-BY-SA-4.0 | {"uuid":"d75c69162cd3ff856429df3ed33ece3d","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL202616"}]} | tier2_rdkit | {"abs_tol":0.000001,"rule":"abs_window"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Compute the quantitative estimate of drug-likeness (QED) of the following SMILES.\n\nCc1cc2cc(C(=O)NCCOC(N)=O)oc2cc1C\n\nAt the end of your response, state the answer in the following format: 'Correct answer is floating_point_number'. The final… | 0.83335 | float | Correct answer is\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | qed | direct | CC-BY-SA-4.0 | {"uuid":"b1f887026acd00ad5da4de7aaaa8eeb4","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL4544835"}]} | tier2_rdkit | {"abs_tol":0.02,"rule":"abs_window"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Report the molecular weight (in g/mol, using average atomic masses) for this compound.\n\nO=P(c1ccccc1)(c1ccccc1)c1ccc2c(c1)OCCOCCOCCOCCO2\n\nConclude your response with: Correct Answer >> floating_point_number. The final result should be a sin… | 468.486 | float | Correct Answer >>\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | MolWt | direct | CC-BY-SA-4.0 | {"uuid":"36095f6b1ce19db2a73675d2081bd454","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL1999765"}]} | tier1_rdkit | {"rel_tol":0.000030000000000000004,"rule":"rel_window"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"What is the count of carbonyl oxygens, excluding COOH (C=O, not COOH) in this molecule?\n\nCCOC(=O)NCCCC(=O)Nc1ccc(O[Si](C)(C)C(C)(C)C)cc1\n\nEnd your response with Final Answer: ||integer||, where integer is the final answer. Do not modify the… | 2 | float | Final Answer:\s*\*{0,2}\|{0,2}\s*([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_C_O_noCOO | direct | CC-BY-SA-4.0 | {"uuid":"8a32873cf30e49b701674158c20c7996","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL4443457"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Calculate the QED of the following SMILES. Give a numeric value.\n\nClc1sc(Cl)c(Cl)c1Cl\n\nProvide your final answer at the end of the response as: Answer: *floating_point_number*, where floating_point_number is the answer. The final result sho… | 0.6109826 | float | Answer:\s*\*{0,2}\*?\s*([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | qed | direct | CC-BY-SA-4.0 | {"uuid":"9e20d78a7288a51d802e9966883f8af1","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL443394"}]} | tier2_rdkit | {"abs_tol":0.02,"rule":"abs_window"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"What is the quantitative estimate of drug-likeness (QED) for the molecule shown?\n\nN#C[C@H]1CCCC[C@@H]1n1nc(Nc2cccc(CNC(=O)c3cnco3)c2)c2c(=O)[nH]ccc21\n\nPut the correct answer at the end like this: Correct Answer: [floating_point_number]. The… | 0.40116477 | float | Correct Answer:\s*\*{0,2}\[?\s*([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | qed | direct | CC-BY-SA-4.0 | {"uuid":"c56589e966e5ab3d53e90c7ed0d87c36","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL5814098"}]} | tier2_rdkit | {"abs_tol":0.02,"rule":"abs_window"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"What is the exact monoisotopic molecular mass (in Da) for the following molecule? Tell me as a decimal number.\n\nCOc1ccc(C2=NN(c3ccccc3)[C@H](c3cn(-c4ccccc4)nc3-c3ccc(F)cc3)C2)cc1\n\nPlace the final answer within \\boxed{}. Do not modify the r… | 488.20123 | float | \\boxed\{\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | ExactMolWt | direct | CC-BY-SA-4.0 | {"uuid":"8c2ef33f686df94517c1356453a74f0d","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL4973436"}]} | tier1_rdkit | {"rel_tol":0.003,"rule":"rel_window"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Identify the number of benzene rings (c6h6) in the following compound. Provide a whole number.\n\nCC(C)c1cc(C(C)C)c(S(=O)(=O)N/N=C2\\CC3CCC2(C)C3(C)C)c(C(C)C)c1\n\nYour final answer must be enclosed in double parentheses. Do not modify the requ… | 1 | float | \(\(\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | fr_benzene | direct | CC-BY-SA-4.0 | {"uuid":"b23c4181da29225c06c435bc1371d21b","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL1994529"}]} | tier2_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"What is the count of six-membered rings in this molecule? Provide a whole number.\n\nCc1cccc(-c2ccc3c(c2)CCC3N[C@@H]2CC[C@@](C(=O)N3CCc4ccc(C(F)(F)F)cc4C3)(C(C)C)C2)c1\n\nYour final answer must be enclosed in double curly braces. Do not modify … | 4 | float | \{\{\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | Num6MemberRings | direct | CC-BY-SA-4.0 | {"uuid":"e22c5bbb1637c02844452eb7e9e06894","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL3417219"}]} | tier1_rdkit | {"rule":"exact"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Report the topological polar surface area (TPSA, in Ų) of the following SMILES. Tell me as a decimal number.\n\nCCN(C(=O)N(C)C)c1cc2oc(-c3ccc(F)cc3)c(C(=O)NC)c2cc1-c1ccc(OC)c(C(=O)NC2(c3ncccn3)CC2)c1\n\nEnd your response with [Answer: floating… | 129.9 | float | \[?Answer:\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | TPSA | direct | CC-BY-SA-4.0 | {"uuid":"3e863efd8ac1fa0d85c487698580c958","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL4571040"}]} | tier2_rdkit | {"abs_tol":0.000001,"rule":"abs_window"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Calculate the topological polar surface area (TPSA, in Ų) for this compound.\n\nCN[C@@H]1C[C@H](c2ccc(Cl)c(Cl)c2)c2ccccc21\n\nPlace the final answer within \\boxed{}. Do not modify the required answer format with any markdown, code fences, quo… | 12.03 | float | \\boxed\{\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | TPSA | direct | CC-BY-SA-4.0 | {"uuid":"d932cdb7d0f64dd736aae3344b465c80","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL1394934"}]} | tier2_rdkit | {"abs_tol":0.000001,"rule":"abs_window"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Calculate the quantitative estimate of drug-likeness (QED) for this compound.\n\nO=C(O)c1cc(F)c(F)cc1-c1ccc(CCc2nc(CC3CCCCC3)c[nH]2)cc1\n\nYour final answer must be enclosed in double curly braces. Do not modify the required answer format with … | 0.49169144 | float | \{\{\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | qed | direct | CC-BY-SA-4.0 | {"uuid":"0cc9fd8a40673266495023911c991150","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL1092730"}]} | tier2_rdkit | {"abs_tol":0.02,"rule":"abs_window"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"What is the predicted Crippen logP (octanol–water partition coefficient) for the molecule shown?\n\nCC(C)Nc1cc(-c2ccc3cc(C#N)cnn23)ncc1-c1nnc(N2CCC3(CC2)COC3)s1\n\nEnd with your answer wrapped like this: [[floating_point_number]]. The final res… | 4.22358 | float | \[\[\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | MolLogP | direct | CC-BY-SA-4.0 | {"uuid":"6885ad698605421ad390d002221a5be4","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL5788603"}]} | tier2_rdkit | {"abs_tol":0.07,"rule":"abs_window"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Determine the topological polar surface area (in Ų) of the following SMILES.\n\nCN1CCN(c2ccc(-c3cc(-c4ccccc4[N+](=O)[O-])ncn3)cc2)CC1\n\nYour final answer should be on a new line as 'Final value is: floating_point_number'. The final result sho… | 75.4 | float | Final value is:\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | TPSA | direct | CC-BY-SA-4.0 | {"uuid":"2aa61682e1b5462620c97c7e19c17ba0","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL5591907"}]} | tier2_rdkit | {"abs_tol":0.000001,"rule":"abs_window"} |
| litmus_bench_v0.2 | {"name":"litmus_agent_agent"} | {"input":[{"role":"user","content":"Determine the Crippen logP for the molecule shown. Give a numeric value.\n\nCCc1ccc(CNC(=O)C2CCN(C(=O)c3ccc(-n4c(C)cc5ccccc54)s3)CC2)cc1\n\nEnd with your answer wrapped like this: [[floating_point_number]]. The final result should be a single f… | 5.73142 | float | \[\[\s*\*{0,2}([-+]?\d*\.?\d+(?:[eE][-+]?\d+)?) | MolLogP | direct | CC-BY-SA-4.0 | {"uuid":"744b0a2a5d240c3b3cdb42e8b0037a55","sources":[{"dataset":"chembl","dataset_version":"36","row_id":"CHEMBL1590035"}]} | tier2_rdkit | {"abs_tol":0.07,"rule":"abs_window"} |
