Dataset
First 50 rows of test
| idstring | familystring | stateJson | question | labelsList | expectedJson | trap_answerJson | groupstring | provenance | tierstring | languagestring | use_casestring | topicstring | label_sourcestring | panel | target_probsJson |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| jb16-llm_router-9a24706d03 | llm_router | Here are three expense lines from my trip. Turn them into a JSON list with fields merchant, date, amount_usd. - Blue Cafe, 2024-03-02, $14.50 - MetroCab, 2024-03-02, $23.00 - Hotel Lira, 2024-03-04, $189.90 | {"type":"choice","instructions":"Classify the task type of this request for routing.","criteria":{"chitchat":"Casual small talk, no task.","creative_writing":"Compose original text.","extraction_json":"Pull given details into a structured JSON shape.","data_analysis":"Compute sta… | ["chitchat","creative_writing","extraction_json","data_analysis"] | extraction_json | data_analysis | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user gives explicit lines and asks for them as JSON with named fields; this is structured extraction, not computing any statistic over them."} | easy | en | model_routing | finance_commerce | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"extraction_json","gptoss":"extraction_json","q38":"extraction_json"},"mean":[0.0001,0,0.9999,0],"status":"unanimous"} | ||
| jb16-llm_router-3be050a02f |
| Can you put this sentence into natural French for a greeting card: 'Thinking of you today and always.' |
| {"type":"choice","instructions":"Classify the task type of this request for routing.","criteria":{"coding":"The request asks to write, fix or explain program code.","creative_writing":"The request asks to compose original prose or verse.","translation":"The request asks to render… |
| ["coding","creative_writing","translation","chitchat"] |
| translation |
| creative_writing |
| {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user gives a fixed sentence and asks for it in French, which is translation; they are not asking for new original writing."} |
| easy |
| en |
| model_routing |
| everyday_language |
| author+panel |
| {"models":["glm47","gptoss","q38"],"answers":{"glm47":"translation","gptoss":"translation","q38":"translation"},"mean":[0,0,1,0],"status":"unanimous"} |
| jb16-llm_router-9bb828d310 | llm_router | Write a short, warm four-line poem I can read at my grandma's 80th birthday. | {"type":"choice","instructions":"Classify the task type of this request for routing.","criteria":{"extraction_json":"Pull given details into structured fields.","translation":"Render given text into another language.","coding":"Write or fix program code.","creative_writing":"Comp… | ["extraction_json","translation","coding","creative_writing"] | creative_writing | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"Composing an original poem is creative writing; there is no source text to translate or extract and no code involved."} | easy | en | model_routing | everyday_language | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"creative_writing","gptoss":"creative_writing","q38":"creative_writing"},"mean":[0,0,0,1],"status":"unanimous"} |
| jb16-llm_router-20fd4a2fe4 | llm_router | Pretty-print this JSON for me: {"id":7,"tags":["a","b"],"ok":true} | {"type":"choice","instructions":"Pick the model tier the router should send this request to.","criteria":{"nano":"Trivial formatting or lookup with one clear output.","standard":"Everyday explanation or simple code, no investigation.","pro":"Investigation or analysis across sever… | ["nano","standard","pro","deep"] | nano | standard | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"Reformatting a tiny JSON object is pure formatting with one correct output, which the smallest tier handles."} | easy | en | model_routing | coding | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"nano","gptoss":"nano","q38":"nano"},"mean":[0.9999,0.0001,0,0],"status":"unanimous"} |
| jb16-llm_router-f6fa427bed | llm_router | Write a query against our Postgres `orders` table that returns total revenue per month for 2024. | {"type":"choice","instructions":"Pick the specialist agent that should handle this request.","criteria":{"python_agent":"Writes or runs standalone Python scripts and data code.","sql_agent":"Writes and runs database queries (SQL).","docs_agent":"Writes or updates documentation an… | ["python_agent","sql_agent","docs_agent","human_review"] | sql_agent | python_agent | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The request is plainly to author a SQL query against a named table, which is the SQL agent's job; no Python, docs or human judgment is involved."} | easy | en | model_routing | coding | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"sql_agent","gptoss":"sql_agent","q38":"sql_agent"},"mean":[0.0001,0.9999,0,0],"status":"unanimous"} |
| jb16-llm_router-00ad4c203d | llm_router | {"conversation":[{"role":"user","content":"morning! hope you had a good weekend :) just saying hi before standup"}]} | {"type":"choice","instructions":"Classify the task type of the latest user turn for routing.","criteria":{"coding":"A request to write or fix code.","chitchat":"Friendly small talk with no task to perform.","data_analysis":"A request to compute over or interpret data.","extractio… | ["coding","chitchat","data_analysis","extraction_json"] | chitchat | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The message is a greeting with no task of any kind, so it is small talk; nothing asks for code, analysis or extraction."} | easy | en | model_routing | everyday_language | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"chitchat","gptoss":"chitchat","q38":"chitchat"},"mean":[0,1,0,0],"status":"unanimous"} |
| jb16-llm_router-c11a8bcce3 | llm_router | A recipe for 4 people uses 300 g of flour. How much flour do I need for 6 people? | {"type":"choice","instructions":"Pick the model tier the router should send this request to.","criteria":{"tier_mini":"A single fact or one trivial step.","tier_core":"Everyday reasoning with a couple of plain steps.","tier_max":"Long, careful multi-step analysis or proofs."}} | ["tier_mini","tier_core","tier_max"] | tier_core | tier_mini | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"Scaling 300 g by 6/4 to 450 g is a short two-step proportion, more than a trivial lookup but far short of deep analysis, so the middle tier fits."} | easy | en | model_routing | math | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"tier_core","gptoss":"tier_mini","q38":"tier_core"},"mean":[0.1671,0.8328,0.0001],"status":"majority"} |
| jb16-llm_router-35eebd3244 | llm_router | Jeg fikk en SMS om at pakken min står fast i tollen, og at jeg må betale 29 kroner via en lenke for at den skal bli sendt videre. Er dette svindel? | {"type":"choice","instructions":"Hvilken assistent skal ruteren sende meldingen til?","criteria":{"parcel_tracking":"Henter status for en kjent forsendelse ut fra et sporingsnummer.","fraud_advice":"Spørsmål om mulig svindel, falske lenker eller nettfisking.","payments_agent":"Ut… | ["parcel_tracking","fraud_advice","payments_agent","small_talk"] | fraud_advice | parcel_tracking | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user asks whether an SMS that demands a small fee through a link is a scam. That is a question about possible fraud and fake links. No tracking number is … | easy | no | model_routing | safety_security | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"fraud_advice","gptoss":"fraud_advice","q38":"fraud_advice"},"mean":[0,0.9999,0,0],"status":"unanimous"} |
| jb16-llm_router-0f763e2a49 | llm_router | Attached: master_services_agreement.pdf (about 210,000 tokens). User: 'List every clause that lets either party terminate the agreement.' model,context_window_tokens,price_per_1m_usd Ondrel-L,512000,2.40 Ondrel-S,32000,0.15 Ondrel-M,128000,0.70 | {"type":"choice","instructions":"Which is the cheapest model whose context window can hold the whole attached document?","criteria":{"ondrel_l":"Ondrel-L.","ondrel_s":"Ondrel-S.","ondrel_m":"Ondrel-M."}} | ["ondrel_l","ondrel_s","ondrel_m"] | ondrel_l | ondrel_m | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The document is about 210,000 tokens. Ondrel-S (32,000) and Ondrel-M (128,000) are too small; only Ondrel-L (512,000) can hold it, so it is the cheapest model… | easy | en | model_routing | law_policy | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"ondrel_l","gptoss":"ondrel_l","q38":"ondrel_l"},"mean":[1,0,0],"status":"unanimous"} |
| jb16-llm_router-3d120e0282 | llm_router | Hvor mye er 1 500 kroner i euro med dagens kurs? | {"type":"choice","instructions":"Hvilket verktøy må ruteren koble til for at assistenten skal kunne svare?","criteria":{"arithmetic_only":"Alle tallene som trengs, står i spørsmålet; det gjenstår bare regning.","none_needed":"Svaret trenger verken tall eller oppslag.","exchange_r… | ["arithmetic_only","none_needed","exchange_rate_feed"] | exchange_rate_feed | arithmetic_only | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user asks for a conversion at today's exchange rate, and no rate is given in the question. So the assistant needs a current exchange rate, which only the … | easy | no | model_routing | math | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"exchange_rate_feed","gptoss":"exchange_rate_feed","q38":"exchange_rate_feed"},"mean":[0,0,1],"status":"unanimous"} |
| jb16-llm_router-a009368474 | llm_router | Get the name, job title and email out of this signature and give it to me as JSON: Best, Marta Olsson Head of Procurement, Brightfield Foods marta.olsson@brightfield-foods.example +46 70 555 01 23 | {"type":"choice","instructions":"This message was typed into the assistant panel of a sales CRM. Which CRM skill should the router call?","criteria":{"outreach_composer":"Writes a new first-contact or follow-up email to a prospect.","lead_scorer":"Rates how likely a company is to… | ["outreach_composer","lead_scorer","pipeline_report","contact_parser","small_talk"] | contact_parser | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user pastes an email signature and wants the name, title and email returned as JSON. Turning pasted contact details into structured fields is the contact … | easy | en | model_routing | everyday_language | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"contact_parser","gptoss":"contact_parser","q38":"contact_parser"},"mean":[0,0,0,1,0],"status":"unanimous"} |
| jb16-llm_router-85cb77c3ea | llm_router | Channel: pharmacy app chat Customer: verified, 2 active prescriptions Message: I forgot my blood pressure pill this morning. Should I just take two tonight to make up for it? | {"type":"choice","instructions":"Which path should the pharmacy app's router choose?","criteria":{"refill_ordering":"Orders a repeat prescription or checks when a refill is ready.","opening_hours_bot":"Opening times and holiday hours of the pharmacy branches.","store_locator":"Fi… | ["refill_ordering","opening_hours_bot","store_locator","dosing_safety_path"] | dosing_safety_path | refill_ordering | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user asks whether to double a dose after a missed pill. That is a missed-dose question, which the dosing safety path covers; the user does not ask for a r… | easy | en | model_routing | support_ops | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"dosing_safety_path","gptoss":"dosing_safety_path","q38":"dosing_safety_path"},"mean":[0,0,0,1],"status":"unanimous"} |
| jb16-llm_router-9963c36ef3 | llm_router | This should return the largest number, but max_of([-3, -7]) gives 0. What's wrong? def max_of(xs): m = 0 for x in xs: if x > m: m = x return m | {"type":"choice","instructions":"Which coding sub-agent should take this request?","criteria":{"test_writer":"Writes new automated tests for given code.","refactorer":"Restructures or renames code without changing what it does.","doc_writer":"Writes comments, docstrings or docume… | ["test_writer","refactorer","doc_writer","bug_hunter"] | bug_hunter | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The function gives a wrong result (0 instead of -3) because m starts at 0, which is larger than every negative input. Finding and fixing the cause of a wrong … | easy | en | model_routing | coding | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"bug_hunter","gptoss":"bug_hunter","q38":"bug_hunter"},"mean":[0,0,0,0.9999],"status":"unanimous"} |
| jb16-llm_router-b7dabe873b | llm_router | Avaa repositorio pajala/laskutus, aja testit, korjaa epäonnistuva testi ja tee korjauksesta uusi haara. | {"type":"choice","instructions":"Mille agentille reitittimen pitää ohjata tämä pyyntö?","criteria":{"code_explainer":"Selittää keskusteluun liitettyä koodia; ei pääse repositorioihin eikä voi ajaa komentoja.","repository_agent":"Kloonaa repositorion, ajaa testejä ja komentoja, mu… | ["code_explainer","repository_agent","translator","spreadsheet_agent"] | repository_agent | code_explainer | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user asks to open a repository, run its tests, fix a failing test and create a branch. Only the repository agent can open repositories, run commands and c… | easy | fi | model_routing | coding | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"repository_agent","gptoss":"repository_agent","q38":"repository_agent"},"mean":[0.0001,0.9999,0,0],"status":"unanimous"} |
| jb16-llm_router-46b508f3d5 | llm_router | what does `git stash` do? | {"type":"choice","instructions":"Which model tier should the router pick for this request?","criteria":{"sprout":"One-line factual answer about a well-known command, term or fact; no code to write.","sapling":"Write or explain a short piece of code (up to about 30 lines) with no … | ["sprout","sapling","oak","sequoia"] | sprout | sapling | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user asks what one well-known git command does. A one-line factual answer is enough and no code has to be written, which is exactly the sprout definition.… | easy | en | model_routing | coding | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"sprout","gptoss":"sprout","q38":"sprout"},"mean":[1,0,0,0],"status":"unanimous"} |
| jb16-llm_router-b75324bc2b | llm_router | Write a short, funny birthday poem for my coworker Dev, who spends all day fighting Rust compiler errors. | {"type":"choice","instructions":"The workplace assistant has four handlers. Which one should take this message?","criteria":{"code_writer":"Writes new program code from a specification.","compiler_error_fixer":"Explains and fixes compiler or build errors in pasted code.","trivia_… | ["code_writer","compiler_error_fixer","trivia_answerer","verse_writer"] | verse_writer | compiler_error_fixer | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user wants a birthday poem. The compiler errors are only the topic of the joke; no code has to be written or fixed, so the verse writer fits."} | easy | en | model_routing | support_ops | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"verse_writer","gptoss":"verse_writer","q38":"verse_writer"},"mean":[0.0001,0,0,0.9998],"status":"unanimous"} |
| jb16-llm_router-b77ca12b8a | llm_router | Ahoj, potřebuju v Pythonu funkci, která vrátí faktoriál čísla n. Stačí krátký kód, vysvětlení nepotřebuju. | {"type":"choice","instructions":"Na kterou úroveň modelu má router tento požadavek poslat?","criteria":{"nimble":"Krátké samostatné úlohy: jedna jednoduchá funkce, krátké vysvětlení nebo jednoduchý fakt.","steady":"Středně velké úlohy: více funkcí nebo celý modul, delší text či e… | ["nimble","steady","summit"] | nimble | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user wants one short Python function (factorial) and explicitly no explanation. One simple function is exactly the nimble definition; there is no module a… | easy | cs | model_routing | coding | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"nimble","gptoss":"nimble","q38":"nimble"},"mean":[1,0,0],"status":"unanimous"} |
| jb16-llm_router-1e088bd700 | llm_router | Lumenbox help chat: 'Someone just logged into my account from another country and changed my email address. I can't sign in anymore. How do I get my account back?' | {"type":"choice","instructions":"Which Lumenbox help desk should the router send this chat to?","criteria":{"billing_desk":"Charges, receipts, payment methods and plan prices.","plan_features_bot":"Explains what each Lumenbox plan includes and how features work.","account_recover… | ["billing_desk","plan_features_bot","account_recovery_desk","feedback_inbox"] | account_recovery_desk | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"A stranger logged in and changed the login email, and the user is locked out. That is a takeover of the account, which the account recovery desk handles; no c… | easy | en | model_routing | support_ops | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"account_recovery_desk","gptoss":"account_recovery_desk","q38":"account_recovery_desk"},"mean":[0,0,1,0],"status":"unanimous"} |
| jb16-llm_router-5445397e8f | llm_router | {"conversation":[{"role":"user","content":"good morning!! how's your day going? 😊"}]} | {"type":"choice","instructions":"Pick the tier for the reply.","criteria":{"chorus_tier":"Hard multi-step reasoning, long documents or complex code.","voice_tier":"Ordinary tasks: emails, explanations, short code.","murmur_tier":"Greetings, thanks and small talk; no task."}} | ["chorus_tier","voice_tier","murmur_tier"] | murmur_tier | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The message is a greeting and small talk. It contains no task, which matches the murmur tier definition."} | easy | en | model_routing | everyday_language | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"murmur_tier","gptoss":"murmur_tier","q38":"murmur_tier"},"mean":[0.0003,0.0001,0.9996],"status":"unanimous"} |
| jb16-llm_router-17f813dcbd | llm_router | Show rigorously that the square root of 3 is irrational. | {"type":"choice","instructions":"How should the router handle this math request?","criteria":{"quick_answer":"A single fact or small arithmetic result that the user just wants stated.","step_by_step_tutor":"The user wants a method explained step by step so that someone can learn … | ["quick_answer","step_by_step_tutor","proof_engine"] | proof_engine | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user asks to show rigorously that a statement is true, which is a request for a mathematical proof. Nothing asks only for the result or for a learning wal… | easy | en | model_routing | math | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"proof_engine","gptoss":"proof_engine","q38":"proof_engine"},"mean":[0,0,1],"status":"unanimous"} |
| jb16-llm_router-cc7b23d004 | llm_router | How do you say 'Where is the nearest pharmacy?' in Portuguese? | {"type":"choice","instructions":"The travel app's assistant has four skills. Which one should the router call?","criteria":{"grammar_coach":"Points out grammar mistakes in a sentence the traveller wrote, in the same language.","phrase_translator":"Gives a phrase in another langua… | ["grammar_coach","phrase_translator","itinerary_summarizer","booking_extractor"] | phrase_translator | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user wants an English sentence said in Portuguese, which is giving a phrase in another language. Nothing is corrected, summarized or extracted from a book… | easy | en | model_routing | everyday_language | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"phrase_translator","gptoss":"phrase_translator","q38":"phrase_translator"},"mean":[0,1,0,0],"status":"unanimous"} |
| jb16-llm_router-2999891ef6 | llm_router | Add type hints to this function: def scale(values, factor): return [v * factor for v in values] | {"type":"choice","instructions":"Which coding tier should handle this?","criteria":{"micro_coder":"Small edits to a few lines of given code: renames, type hints, formatting.","full_coder":"New features or modules, or changes that touch several files.","architect":"System design a… | ["micro_coder","full_coder","architect"] | micro_coder | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"Adding type hints to a two-line function is a small edit to given code. The micro coder definition names type hints explicitly."} | easy | en | model_routing | coding | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"micro_coder","gptoss":"micro_coder","q38":"micro_coder"},"mean":[1,0,0],"status":"unanimous"} |
| jb16-llm_router-1f3f35f724 | llm_router | Here are our delivery times in minutes for last week: 32, 41, 28, 35, 50, 29, 37. What are the mean and the median? | {"type":"choice","instructions":"Which math tool should the router attach to this request?","criteria":{"calculator":"Evaluates a single numeric arithmetic expression with no variables.","symbolic_solver":"Works with expressions that contain variables: solving, simplifying, diffe… | ["calculator","symbolic_solver","statistics_engine","unit_converter"] | statistics_engine | calculator | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user wants the mean and the median of a list of seven numbers. These are summary statistics over a list, which the statistics engine is defined to compute… | easy | en | model_routing | math | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"statistics_engine","gptoss":"statistics_engine","q38":"statistics_engine"},"mean":[0.0001,0,0.9999,0],"status":"unanimous"} |
| jb16-llm_router-97601d996d | llm_router | Dinner came to $184.50. Add an 18% tip and split the total evenly among 6 people. How much does each person pay? | {"type":"choice","instructions":"Which tool should the router attach to this request?","criteria":{"calculator":"Arithmetic on numbers that are all given in the request.","no_tool":"No arithmetic, lookup or scheduling is involved.","web_search":"Needs current information from the… | ["calculator","no_tool","web_search","calendar"] | calculator | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"All numbers are in the request ($184.50, 18%, 6 people), so only arithmetic is left: 184.50 × 1.18 = 217.71, and 217.71 / 6 = 36.285, so about $36.29 each. No… | easy | en | model_routing | math | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"calculator","gptoss":"calculator","q38":"calculator"},"mean":[1,0,0,0],"status":"unanimous"} |
| jb16-llm_router-6726b5c405 | llm_router | Kettleworks web chat: 'Hi, where is my order #KX-20417? The tracking email said it would arrive on Monday.' | {"type":"choice","instructions":"Which Kettleworks support bot should the router send this chat to?","criteria":{"courier_status_bot":"Reads the courier feed for an order Kettleworks has already shipped and reports where it is and the new delivery estimate.","damaged_goods_desk":… | ["courier_status_bot","damaged_goods_desk","presale_advisor","legal_threats_queue"] | courier_status_bot | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The customer asks where an order that has already shipped is and when it will arrive. That is a courier status lookup; nothing is broken, nothing is bought ye… | easy | en | model_routing | support_ops | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"courier_status_bot","gptoss":"courier_status_bot","q38":"courier_status_bot"},"mean":[1,0,0,0],"status":"unanimous"} |
| jb16-llm_router-a6eea1cea1 | llm_router | Please review this diff before I merge it: - if user.is_admin: + if user.is_admin or user.email.endswith('@corp.example'): grant_full_access(user) | {"type":"choice","instructions":"The engineering assistant has four handlers. Which one should take this request?","criteria":{"feature_builder":"Writes new code for a feature from a description.","standup_chat":"Casual team chat with no engineering task.","readme_editor":"Edits … | ["feature_builder","standup_chat","readme_editor","pr_reviewer"] | pr_reviewer | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user pastes a diff and asks for a review before merging. Reviewing a proposed diff is the PR reviewer's job; no new feature is requested and no documentat… | easy | en | model_routing | coding | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"pr_reviewer","gptoss":"pr_reviewer","q38":"pr_reviewer"},"mean":[0,0,0,1],"status":"unanimous"} |
| jb16-llm_router-052289d2f2 | llm_router | What is the default port number for HTTPS? | {"type":"choice","instructions":"Which tier should the router choose?","criteria":{"mid_writer":"Explanations, emails and code that need a few paragraphs or a short snippet.","cheap_lookup":"A single well-known fact with a one-line answer.","expert_reasoner":"Multi-step analysis,… | ["mid_writer","cheap_lookup","expert_reasoner"] | cheap_lookup | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user asks for one well-known fact (port 443). A one-line answer is enough, which matches cheap_lookup."} | easy | en | model_routing | coding | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"cheap_lookup","gptoss":"cheap_lookup","q38":"cheap_lookup"},"mean":[0.0001,0.9999,0],"status":"unanimous"} |
| jb16-llm_router-016257c513 | llm_router | Find the derivative of f(x) = x^3 · sin(x). | {"type":"choice","instructions":"Which math tool should the router attach to this request?","criteria":{"calculator":"Evaluates a single numeric arithmetic expression with no variables.","symbolic_solver":"Works with expressions that contain variables: solving, simplifying, diffe… | ["calculator","symbolic_solver","statistics_engine","unit_converter"] | symbolic_solver | calculator | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"Differentiating an expression with the variable x is symbolic work, which the symbolic solver covers. There is no plain number expression, no list of data and… | easy | en | model_routing | math | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"symbolic_solver","gptoss":"symbolic_solver","q38":"symbolic_solver"},"mean":[0,1,0,0],"status":"unanimous"} |
| jb16-llm_router-cf5c8f46c6 | llm_router | Kirjoita lyhyt ja lämmin runo äidilleni äitienpäiväksi. Hän rakastaa puutarhanhoitoa. | {"type":"choice","instructions":"Henkilökohtaisella avustajalla on neljä taitoa. Mihin niistä reitittimen pitää ohjata tämä pyyntö?","criteria":{"fact_finder":"Vastaa yksittäiseen tietokysymykseen lyhyesti.","translator":"Kääntää käyttäjän tekstin toiselle kielelle.","verse_write… | ["fact_finder","translator","verse_writer","form_filler"] | verse_writer | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user asks for a short new poem for their mother for Mother's Day. Writing a new poem is the verse writer's skill; nothing is looked up, translated or extr… | easy | fi | model_routing | everyday_language | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"verse_writer","gptoss":"verse_writer","q38":"verse_writer"},"mean":[0,0,1,0],"status":"unanimous"} |
| jb16-llm_router-d21e6ca2dc | llm_router | How do I reverse a string in Python? Just need the one-liner. | {"type":"choice","instructions":"Pick the model tier the router should send this request to.","criteria":{"small_fast":"Short factual or formatting tasks with a single obvious answer.","general_mid":"Everyday explanations or simple code that needs no investigation.","frontier":"M… | ["small_fast","general_mid","frontier","frontier_reasoning"] | small_fast | general_mid | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"Reversing a string is a single well-known one-liner with one obvious answer, so the smallest tier handles it; there is nothing to investigate or reason throug… | easy | en | model_routing | coding | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"small_fast","gptoss":"small_fast","q38":"general_mid"},"mean":[0.6667,0.3333,0,0],"status":"majority"} |
| jb16-llm_router-0883e79c52 | llm_router | Book a 30-minute call with Priya Nair next Tuesday at 3 pm and send her an invite. | {"type":"choice","instructions":"Which tool should the router give the assistant for this request?","criteria":{"web_search":"Looks up public information online.","no_tool":"The answer needs no action and no lookup.","email_drafting_only":"Writes email text for the user to send t… | ["web_search","no_tool","email_drafting_only","calendar_tool"] | calendar_tool | email_drafting_only | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user wants a calendar event created and an invite sent. Creating events and sending their invites is what the calendar tool does; drafting an email would … | easy | en | model_routing | everyday_language | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"calendar_tool","gptoss":"calendar_tool","q38":"calendar_tool"},"mean":[0,0,0,1],"status":"unanimous"} |
| jb16-llm_router-203873a445 | llm_router | My Android app crashes when the phone rotates while the camera screen is open. It's written in Kotlin; logcat output is below. [30 lines of logcat] | {"type":"choice","instructions":"Which specialist should the router assign?","criteria":{"frontend_specialist":"Browser UI code: React components, CSS, HTML and client-side JavaScript.","backend_specialist":"Server APIs, databases, queues and other server-side services.","mobile_… | ["frontend_specialist","backend_specialist","mobile_specialist","data_specialist"] | mobile_specialist | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The crash happens in a native Android phone app, so the mobile specialist is the match. It is not browser code, a server service or a data task."} | easy | en | model_routing | coding | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"mobile_specialist","gptoss":"mobile_specialist","q38":"mobile_specialist"},"mean":[0,0,1,0],"status":"unanimous"} |
| jb16-llm_router-9f05a595bc | llm_router | Write pytest unit tests for this function: def is_leap(year): return year % 4 == 0 and (year % 100 != 0 or year % 400 == 0) | {"type":"choice","instructions":"Which coding sub-agent should take this request?","criteria":{"test_writer":"Writes new automated tests for given code.","refactorer":"Restructures or renames code without changing what it does.","doc_writer":"Writes comments, docstrings or docume… | ["test_writer","refactorer","doc_writer","bug_hunter"] | test_writer | bug_hunter | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user asks for new unit tests for a given function and reports no bug. Writing new automated tests is the test writer's job."} | easy | en | model_routing | coding | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"test_writer","gptoss":"test_writer","q38":"test_writer"},"mean":[1,0,0,0],"status":"unanimous"} |
| jb16-llm_router-8968731ec5 | llm_router | hey there, are you a bot? | {"type":"choice","instructions":"Which tier should the router choose?","criteria":{"pocket":"Greetings, yes/no facts, or one-sentence answers; no code.","desk":"Short self-contained code snippets or explanations (one function, one regex, one query).","studio":"Multi-file features… | ["pocket","desk","studio","lab"] | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"This is a greeting plus a yes/no question. It needs a one-sentence answer and no code, which is the pocket tier definition."} | easy | en | model_routing | support_ops | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"pocket","gptoss":"pocket","q38":"pocket"},"mean":[1,0,0,0],"status":"unanimous"} |
| jb16-llm_router-c48a5dfb14 | llm_router | {"conversation":[{"role":"user","content":"According to the employee handbook, how many vacation days do I get once I've been here 3 years?"}],"attachments":[{"name":"employee_handbook_2026.pdf","pages":48}],"workspace":"internal HR assistant"} | {"type":"choice","instructions":"Which internal agent should the router send this question to?","criteria":{"payroll_agent":"Looks up the user's own payslips, salary payments and tax forms.","it_helpdesk_agent":"Laptops, passwords, VPN and software access.","travel_booking_agent"… | ["payroll_agent","it_helpdesk_agent","travel_booking_agent","facilities_agent","handbook_qa_agent"] | handbook_qa_agent | payroll_agent | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user asks what the employee handbook says about vacation days, with the handbook attached. Answering questions about company rules and benefits from the h… | easy | en | model_routing | law_policy | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"handbook_qa_agent","gptoss":"handbook_qa_agent","q38":"handbook_qa_agent"},"mean":[0,0,0,0,1],"status":"unanimous"} |
| jb16-llm_router-c17b8620b1 | llm_router | {"conversation":[{"role":"user","content":"Please write out everything that was said in this recorded support call so I can attach a full transcript to ticket #5521."}],"attachments":[{"name":"support_call_5521.m4a","type":"audio","duration":"00:12:40"}]} | {"type":"choice","instructions":"Which model should the router use for this request?","criteria":{"speech_to_text":"Turns recorded speech into written text.","text_to_speech":"Reads given text aloud as audio.","image_captioner":"Describes what is shown in images.","summarizer":"S… | ["speech_to_text","text_to_speech","image_captioner","summarizer"] | speech_to_text | summarizer | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The input is an audio recording and the user wants a full transcript of everything said. Turning recorded speech into written text is speech-to-text; a summar… | easy | en | model_routing | support_ops | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"speech_to_text","gptoss":"speech_to_text","q38":"speech_to_text"},"mean":[1,0,0,0],"status":"unanimous"} |
| jb16-llm_router-711d5a8644 | llm_router | [ALERT 14:02 UTC] checkout-api 5xx error rate 38% (threshold 5%). On-call, please investigate. Last deploy: 13:55 UTC, build 8841. | {"type":"choice","instructions":"Which agent should this message be routed to?","criteria":{"release_notes_writer":"Writes changelogs and release notes.","sales_assistant":"Quotes, pricing and questions from prospects.","incident_responder":"Live production problems: alerts, outa… | ["release_notes_writer","sales_assistant","incident_responder","customer_reply_writer"] | incident_responder | release_notes_writer | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The message is a live production alert: the checkout API error rate is 38% against a 5% threshold. Error spikes in production belong to the incident responder… | easy | en | model_routing | support_ops | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"incident_responder","gptoss":"incident_responder","q38":"incident_responder"},"mean":[0,0,1,0],"status":"unanimous"} |
| jb16-llm_router-04ed6df846 | llm_router | what's 9 times 7 | {"type":"choice","instructions":"How should the router handle this math request?","criteria":{"quick_answer":"A single fact or small arithmetic result that the user just wants stated.","step_by_step_tutor":"The user wants a method explained step by step so that someone can learn … | ["quick_answer","step_by_step_tutor","proof_engine"] | quick_answer | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user only wants a small arithmetic result stated (63). Nothing asks for an explanation of a method or for a proof."} | easy | en | model_routing | math | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"quick_answer","gptoss":"quick_answer","q38":"quick_answer"},"mean":[1,0,0],"status":"unanimous"} |
| jb16-llm_router-2a2d2f6950 | llm_router | Operator rule R-7: any request that includes patient health records must go to the on-premise model Halden-Local. All other requests go to the cheapest suitable cloud model. Request: 'Summarize the attached discharge notes for patient 4471 into five bullet points for the ward han… | {"type":"choice","instructions":"Under rule R-7, where must the router send this request?","criteria":{"cloud_small":"The cheapest cloud model, for short simple tasks.","halden_local":"The on-premise model Halden-Local.","cloud_large":"The large cloud model, for hard tasks."}} | ["cloud_small","halden_local","cloud_large"] | halden_local | cloud_small | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The attachment is a patient's discharge notes, which are patient health records. Rule R-7 sends every such request to Halden-Local, even though the summary it… | easy | en | model_routing | law_policy | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"halden_local","gptoss":"halden_local","q38":"halden_local"},"mean":[0.0001,0.9999,0],"status":"unanimous"} |
| jb16-llm_router-8d97708066 | llm_router | Can you fix the grammar? 'Me and him was going to the store yesterday.' | {"type":"choice","instructions":"The writing app routes each message to one editing mode. Which mode fits this message?","criteria":{"translate_mode":"Renders the user's text in another human language.","condense_mode":"Cuts a long text down to its main points.","compose_mode":"W… | ["translate_mode","condense_mode","compose_mode","banter_mode","polish_mode"] | polish_mode | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user gives one English sentence and asks for its grammar to be fixed. Correcting grammar while keeping the language and meaning is the polish mode; nothin… | easy | en | model_routing | everyday_language | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"polish_mode","gptoss":"polish_mode","q38":"polish_mode"},"mean":[0,0,0,0,0.9999],"status":"unanimous"} |
| jb16-llm_router-7a5ddf75f9 | llm_router | What does O(n log n) mean, in plain words? | {"type":"choice","instructions":"Which tier should the router choose?","criteria":{"flicker":"Short definition or explanation of one common concept, a few sentences long.","glow":"Writing or reviewing code, or explanations that need worked examples over several steps.","beacon":"… | ["flicker","glow","beacon"] | flicker | glow | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user wants one common concept explained in plain words. A few sentences are enough, which is the flicker definition; no code and no worked multi-step exam… | easy | en | model_routing | coding | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"flicker","gptoss":"flicker","q38":"flicker"},"mean":[1,0,0],"status":"unanimous"} |
| jb16-llm_router-6e121ec7f7 | llm_router | {"request":"Here's a screenshot of the error dialog my app shows on startup. What does it mean and how do I fix it?","attachments":[{"type":"image/png","name":"error_dialog.png"}],"model_pool":[{"name":"Velloc-Mini","vision":false,"price_per_1m_tokens_usd":0.2},{"name":"Velloc-Si… | {"type":"choice","instructions":"Which model is the cheapest one in the pool that can handle this request?","criteria":{"velloc_mini":"Velloc-Mini from the model pool.","velloc_sight":"Velloc-Sight from the model pool.","velloc_text_pro":"Velloc-Text-Pro from the model pool."}} | ["velloc_mini","velloc_sight","velloc_text_pro"] | velloc_sight | velloc_mini | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The request contains an image, so the model needs vision. Only Velloc-Sight has vision, so it is the only and therefore the cheapest model that can handle it,… | easy | en | model_routing | coding | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"velloc_sight","gptoss":"velloc_sight","q38":"velloc_sight"},"mean":[0.0002,0.9998,0],"status":"unanimous"} |
| jb16-llm_router-11dd62ea8e | llm_router | Calculate the monthly payment on a $20,000 car loan at 6% annual interest over 5 years. | {"type":"choice","instructions":"Which finance tool should the router call?","criteria":{"stock_quote_lookup":"Gets current share prices.","tax_rules_lookup":"Finds tax rates and tax rules.","amortization_calculator":"Computes loan payments, interest and payoff schedules from giv… | ["stock_quote_lookup","tax_rules_lookup","amortization_calculator","invoice_generator"] | amortization_calculator | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The request gives loan terms (principal, rate, term) and asks for the monthly payment. Computing loan payments from loan terms is the amortization calculator'… | easy | en | model_routing | finance_commerce | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"amortization_calculator","gptoss":"amortization_calculator","q38":"amortization_calculator"},"mean":[0,0,1,0],"status":"unanimous"} |
| jb16-llm_router-b32c3bd07c | llm_router | main.go panics with 'assignment to entry in nil map' on the second line here: var counts map[string]int counts["apples"]++ | {"type":"choice","instructions":"Which language helper should the router assign?","criteria":{"go_helper":"Go code and Go runtime errors.","python_helper":"Python code and Python tracebacks.","java_helper":"Java code and JVM exceptions.","javascript_helper":"JavaScript and TypeSc… | ["go_helper","python_helper","java_helper","javascript_helper"] | go_helper | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The file is main.go and the code uses Go syntax (var ... map[string]int); the panic is a Go runtime error. The Go helper is the only match."} | easy | en | model_routing | coding | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"go_helper","gptoss":"go_helper","q38":"go_helper"},"mean":[1,0,0,0],"status":"unanimous"} |
| jb16-llm_router-f0da6c5c6d | llm_router | Varför får jag ”IndexError: list index out of range” här? namn = ['Anna', 'Erik'] for i in range(3): print(namn[i]) | {"type":"choice","instructions":"Vilken specialistagent ska routern skicka frågan till?","criteria":{"sql_helper":"Skriver och rättar databasfrågor.","css_helper":"Layout och utseende på webbsidor.","excel_helper":"Formler och funktioner i kalkylblad.","python_debugger":"Hittar o… | ["sql_helper","css_helper","excel_helper","python_debugger"] | python_debugger | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user shows a Python loop that raises IndexError (range(3) on a list with two names). Finding and explaining errors in Python code is the Python debugger's… | easy | sv | model_routing | coding | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"python_debugger","gptoss":"python_debugger","q38":"python_debugger"},"mean":[0,0,0,1],"status":"unanimous"} |
| jb16-llm_router-00f18390e7 | llm_router | Θέλω να νοικιάσω το διαμέρισμά μου σε έναν φοιτητή για έναν χρόνο. Μπορείς να μου γράψεις ένα απλό ιδιωτικό συμφωνητικό μίσθωσης; | {"type":"choice","instructions":"Ο βοηθός νομικών εγγράφων έχει τέσσερις λειτουργίες. Σε ποια πρέπει να στείλει ο δρομολογητής αυτό το αίτημα;","criteria":{"tax_calculation":"Υπολογίζει φόρους ή εισφορές από ποσά που δίνει ο χρήστης.","dispute_advice":"Δίνει γενική καθοδήγηση για… | ["tax_calculation","dispute_advice","chit_chat","contract_drafting"] | contract_drafting | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user asks for a simple one-year lease agreement to be written. Writing a new agreement is contract drafting; there is no existing dispute and no tax amoun… | easy | el | model_routing | law_policy | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"contract_drafting","gptoss":"contract_drafting","q38":"contract_drafting"},"mean":[0,0,0,1],"status":"unanimous"} |
| jb16-llm_router-cf0c3561e3 | llm_router | Can you turn this Python into JavaScript for me? def greet(name): return f"Hello, {name}!" | {"type":"choice","instructions":"The developer help desk router has four handlers. Which one should get this message?","criteria":{"prose_translator":"Translates messages, docs or UI strings from one human language into another.","changelog_digest":"Condenses long commit logs or … | ["prose_translator","changelog_digest","code_porter","watercooler"] | code_porter | prose_translator | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user wants a Python function rewritten in JavaScript, which is moving code between programming languages. No human-language text is translated and nothing… | easy | en | model_routing | coding | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"code_porter","gptoss":"code_porter","q38":"code_porter"},"mean":[0,0,1,0],"status":"unanimous"} |
| jb16-llm_router-7f929af6ff | llm_router | Client: Harlow Design Work: 12 hours of consulting in September Rate: $95/hour Terms: payment due in 30 days User note: turn this into an invoice I can send them | {"type":"choice","instructions":"Which finance tool should the router call?","criteria":{"stock_quote_lookup":"Gets current share prices.","tax_rules_lookup":"Finds tax rates and tax rules.","amortization_calculator":"Computes loan payments, interest and payoff schedules from giv… | ["stock_quote_lookup","tax_rules_lookup","amortization_calculator","invoice_generator"] | invoice_generator | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user wants an invoice document built from billable hours, a rate and payment terms. That is exactly what the invoice generator does."} | easy | en | model_routing | finance_commerce | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"invoice_generator","gptoss":"invoice_generator","q38":"invoice_generator"},"mean":[0,0,0,1],"status":"unanimous"} |
| jb16-llm_router-6ee45f94ae | llm_router | Πώς ενώνω δύο λίστες σε μία στην Python; | {"type":"choice","instructions":"Σε ποιο επίπεδο μοντέλου πρέπει να στείλει ο δρομολογητής αυτό το αίτημα;","criteria":{"balanced_tier":"Συγγραφή ή εξήγηση κώδικα με αρκετές συναρτήσεις, ή κείμενα μεσαίου μήκους.","light_tier":"Σύντομες ερωτήσεις που απαντώνται με μία ή δύο γραμμ… | ["balanced_tier","light_tier","heavy_tier"] | light_tier | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user asks how to join two lists in Python. One line of code (list1 + list2) answers it, which matches the light tier; no several functions or multi-file w… | easy | el | model_routing | coding | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"light_tier","gptoss":"light_tier","q38":"light_tier"},"mean":[0,1,0],"status":"unanimous"} |
| jb16-llm_router-fc800f7b91 | llm_router | {"conversation":[{"role":"user","content":"Open the repo northwind/billing-api, run the test suite, fix whatever fails and push a branch with the fix."}],"account":{"git_host_connected":true,"plan":"team"}} | {"type":"choice","instructions":"Which agent should the router send this request to?","criteria":{"chat_only_model":"Answers from the text in the conversation; cannot open repositories or run commands.","repo_agent":"Can clone repositories, run tests and commands, edit files and … | ["chat_only_model","repo_agent","web_research_agent","image_model"] | repo_agent | chat_only_model | {"source":"OpenDecisions, written by an LLM author, confirmed by the labeler panel","license":"CC BY-SA 4.0","rationale":"The user asks to open a repository, run its tests, edit code and push a branch. Only the repo agent can clone repositories, run commands and push; a chat-only… | easy | en | model_routing | coding | author+panel | {"models":["glm47","gptoss","q38"],"answers":{"glm47":"repo_agent","gptoss":"repo_agent","q38":"repo_agent"},"mean":[0.0001,0.9999,0,0],"status":"unanimous"} |
