Research Lead, Evaluations and Benchmarks

免费注册后解锁"你的匹配分 vs 高分画像"与个人技能缺口。

Description You ship a benchmark every two to three weeks (example benchmark). Each one measures a frontier risk that nobody has measured yet. Some go public. Some go only to the labs. Some of the benchmarks and papers are done in collaboration with the leading AI Labs and universities. You will not write every eval yourself. Each benchmark pairs you with an in-house researcher who owns that harm area, and you get a budget for freelancers you direct. You own the taxonomy, the harness, the quality bar and the release. The seat sits in the CTO office alongside the research lead who sets our public research agenda. Around 150 researchers here work on harms directly, and you can pull any of them onto a subject. Key Responsibilities 1. Ship the benchmark cadence A benchmark every two to three weeks. Size follows the subject. A chat-based taxonomy can carry 100 evals. An agentic or GRPO benchmark is closer to 20, because each one is expensive to read. Sensitivity decides what ships publicly and what goes to the labs alone. 2. Own the quality bar The test is simple. A frontier lab reruns our set and gets our numbers. The verifiers hold, the rubrics are clear, the distribution is sane, and their subject matter experts read the taxonomy and call it novel. That means you read the evals yourself. You can screen with a model, and you still open the file, spot the item that does not match the taxonomy, and push the researcher back on it. 3. Run the process Hold the plan and the calendar. You are the person who keeps other researchers on timeline. You can direct two or three freelancers (SMEs) yourself ad-hoc when needed. 4. Set the roadmap with the forum Roughly monthly you sit with the CTO and the pod and research leads. Inputs are what our research teams see, what clients are asking for, and what is moving in the news. Output is a revised release plan for the quarter, tied to the accounts we want to open. 5. Stay ahead of the curve Around 20% of your time goes to the ecosystem. Read the research, keep contacts inside the labs, ask them what is bothering them, and travel to a couple of conferences a year. You should be talking to folks from the labs weekly. Requirements Requirements PhD or Masters in computer science, machine learning or a related field, or equivalent depth from industry research. 3+ years building and running safety or security evaluations for language models in production, at an AI lab, a model provider, or a safety and security research organisation. 5+ relevant research publications in the field of AI safety and security including lead author on at least 2 of them Strong engineer. Evaluation harnesses, distributed inference, vLLM, reading a codebase and fixing it. You can build a taxonomy, not only score against one. You can direct a researcher and two freelancers without managing them formally. Strong English, written and spoken. This role runs on internal communication across time zones. Curiosity about the harms themselves. You will be learning a new subject every three weeks. Ideally: Post-training experience: SFT, DPO, GRPO. Reward design for subjective and safety-relevant targets is a live problem for us. Agentic evaluation experience: tool use, orchestration, permissions, prompt injection. Publications at top conferences. Willingness to present your own work on a client call. Strong communication - both verbal and written, ability to present to large and/or senior audiences Travel to conferences at least 3 times a year About Alice Alice is a trust, safety, and security company built for the AI era. We safeguard the communicative technologies people use to create, collaborate, and interact- whether with each other or with machines. Alice clients are the top 8 AI Labs in the world. In a world where AI has fundamentally changed the nature of risk, Alice provides end-to-end coverage across the entire AI lifecycle. We support frontier model labs, enterprises, and UGC platforms with a comprehensive suite of solutions

描述 你每两到三周交付一个基准测试（示例基准）。每个基准都衡量一个尚未有人衡量的前沿风险。有些会公开。有些只提供给实验室。部分基准和论文是与领先的AI实验室和大学合作完成的。你不需要自己编写每一个评估。每个基准都会让你与一位拥有该危害领域的内部研究员配对，并且你可以获得指导自由职业者的预算。你负责分类体系、测试框架、质量标准和发布。该职位设在CTO办公室，与研究负责人并肩工作，后者制定我们的公开研究议程。这里约有150名研究员直接从事危害相关研究，你可以将其中任何人调入某个课题。 关键职责 1. 按节奏交付基准 每两到三周一个基准。规模取决于主题。基于聊天的分类体系可以承载100个评估。Agentic或GRPO基准则接近20个，因为每个评估的阅读成本都很高。敏感性决定哪些公开发布，哪些仅提供给实验室。 2. 把控质量标准 测试很简单。前沿实验室重新运行我们的测试集，得到我们的数字。验证器成立，评分标准清晰，分布合理，并且他们的领域专家阅读分类体系后认为它具有新颖性。这意味着你要亲自阅读评估。你可以用模型进行筛选，但你仍然要打开文件，找出与分类体系不匹配的条目，并让研究员回去修改。 3. 运行流程 掌握计划和日历。你是让其他研究员按时推进的人。你可以在需要时临时直接指导两到三名自由职业者（SME）。 4. 与论坛共同制定路线图 大约每月一次，你与CTO、小组和研究负责人坐在一起。输入是我们的研究团队所看到的内容、客户所要求的内容以及新闻中正在发展的内容。输出是修订后的季度发布计划，并与我们想要开拓的客户账户挂钩。 5. 保持领先 你大约20%的时间用于生态系统。阅读研究，保持与实验室内部的联系，询问他们有什么困扰，并且每年参加两到三次会议。你应该每周都与实验室的人交流。 要求 要求 计算机科学、机器学习或相关领域的博士或硕士学位，或来自行业研究的同等深度。 3年以上在AI实验室、模型提供商或安全与安保研究组织中，为生产中的语言模型构建和运行安全或安保评估的经验。 在AI安全与安保领域有5篇相关研究论文，其中至少2篇为第一作者 强大的工程师。评估框架、分布式推理、vLLM、阅读代码库并修复它。你能够构建分类体系，而不仅仅是对照分类体系打分。你能够指导一名研究员和两名自由职业者，而不必对他们进行正式管理。 较强的英语书面和口语能力。该职位依赖跨时区的内部沟通。 对危害本身有好奇心。你每三周都会学习一个新主题。 理想情况下： 后训练经验：SFT、DPO、GRPO。针对主观和安全相关目标的奖励设计对我们来说是一个现实问题。 Agentic评估经验：工具使用、编排、权限、提示注入。 在顶级会议发表论文。 愿意在客户电话中展示自己的工作。 较强的沟通能力——包括口头和书面，能够向大型和/或资深受众展示 每年至少参加3次会议 关于Alice Alice是一家为AI时代打造的信任、安全和安保公司。我们保护人们用于创造、协作和互动的通信技术——无论是彼此之间还是与机器之间。Alice的客户是全球排名前8的AI实验室。在AI从根本上改变了风险性质的世界中，Alice在整个AI生命周期中提供端到端覆盖。我们通过全面的解决方案套件支持前沿模型实验室、企业和UGC平台

⚑ 岗位信息收录自公开招聘页（Himalayas），以雇主官网为准