Data Engineer - Data Platform

免费注册后解锁"你的匹配分 vs 高分画像"与个人技能缺口。

Firmable is the market-leading B2B sales intelligence platform in Asia Pacific — and we're scaling that success globally at pace. Backed by leading investors and 2,000+ customers strong, we exist to give sales teams an unfair advantage: the deepest company and people data of any platform, enriched with real-time signals, served at the right moment by intelligent agents. Our data is the product. Building it now means building with LLMs in the pipeline, and knowing exactly when to trust them. The Role As Data Engineer — Data Platform , you'll build the pipelines that turn billions of raw records into the clean, modelled datasets our product, analytics, and AI systems depend on. Some of that is classic data engineering: SQL, dbt, orchestration, warehouse performance. A growing share is not: LLM-based extraction, enrichment, and validation running inside the pipeline at scale. The hard part isn't calling a model. It's making a probabilistic component behave like a reliable one: labelled eval sets, precision and recall you can measure, versioned prompts, cost ceilings, and the harnesses that catch regressions when a vendor silently changes the model under you. This is a hands-on engineering role with real ownership . You'll own reliability, cost, and whether downstream consumers can trust the data, whether it came from a SQL join or a language model. What You'll Own LLM-in-the-pipeline systems — extraction, enrichment, entity resolution, and semantic validation steps that run over millions of records a day with structured outputs, retries, and human-review fallbacks Eval harnesses — labelled sets, scorers, and regression suites that gate every prompt or model change; precision/recall tracked per check, not vibes Rules vs. LLMs — deterministic checks (dbt tests, data contracts, SQL) wherever structure allows; LLMs where semantic judgement is needed; the discipline to know which is which Cost and drift — token budgets per pipeline, model routing (cheap models for classification, stronger ones for hard cases), drift detection on vendor updates Observability — every LLM call logged with prompt version, model, cost, latency, and decision, alongside standard pipeline alerting that surfaces problems before they cascade Core pipelines and warehouse — Airflow orchestration, dbt models across staging to mart, Snowflake performance and cost over billions of rows, AWS infrastructure as code Matching and deduplication — embeddings and retrieval patterns for company and people entity resolution across 13 markets What We're Looking For Must Haves 3+ years in data engineering or data infrastructure, with production pipelines you've owned end to end and 2+ years experience with Snowflake Strong Python and SQL — production-grade, performance-aware, comfortable at very large scale Shipped LLMs inside data pipelines — extraction, enrichment, or validation in production, with structured outputs and a labelled eval set that tells you where the model gets it wrong Harness engineering experience — you've built or maintained eval or test harnesses for LLM outputs, and you can talk through what they caught Sharp judgement on rules vs. LLMs — you reach for a regex or a dbt test first and can defend the call either way Production dbt and Airflow — modular, tested models; DAGs that recover gracefully Cloud warehouse experience — Snowflake preferred; schema design, query optimisation, cost management AI coding tools are how you work — Claude Code, Cursor, or equivalent, daily, with real shipped work to show for it Ownership and systems thinking — you weigh upstream dependencies and downstream impact before changing anything Highly Valued Eval frameworks and LLM tracing (Logfire, OpenTelemetry) Embeddings, vector search, or fuzzy matching for entity resolution at scale Fine-tuning or distilling small models to replace expensive LLM calls AWS at scale (S3, Lambda, Glue, ECS, RDS) and PostgreSQL Spark or PySpark; streaming (Kafka, Kinesis, Snowpipe Streaming) Data

Firmable 是亚太地区市场领先的 B2B 销售情报平台——我们正以飞快的速度将这一成功扩展至全球。在顶级投资方的支持和 2,000+ 客户的信任下，我们的使命是为销售团队赋予不公平的优势：提供所有平台中最深的公司和个人数据，辅以实时信号，由智能代理在恰当的时机呈现。我们的数据就是产品。现在构建它，意味着在管道中使用 LLM，并确切知道何时该信任它们。

**职位**

作为数据工程师——数据平台，你将构建将数十亿原始记录转化为干净、建模数据集的管道，这些数据集是我们的产品、分析和 AI 系统所依赖的。其中一部分是经典的数据工程：SQL、dbt、编排、仓库性能。越来越多的部分则不是：基于 LLM 的提取、丰富和验证，在管道内大规模运行。困难的部分不是调用模型。而是让一个概率性组件表现得像一个可靠的组件：标注的评估集、可衡量的精确率和召回率、版本化的提示词、成本上限，以及当供应商在你不知情的情况下悄悄更换模型时能够捕获回归的测试框架。

这是一个具有真正所有权的手动工程角色。你将负责可靠性、成本，以及下游消费者能否信任数据——无论数据来自 SQL 连接还是语言模型。

**你将负责**

- **管道中的 LLM 系统**——提取、丰富、实体解析和语义验证步骤，每天在数百万条记录上运行，具备结构化输出、重试和人工审核回退机制
- **评估框架**——标注集、评分器和回归测试套件，为每一次提示词或模型变更把关；按检查项跟踪精确率/召回率，而非凭感觉
- **规则 vs. LLM**——在结构允许的地方使用确定性检查（dbt 测试、数据契约、SQL）；在需要语义判断的地方使用 LLM；以及区分两者的纪律
- **成本与漂移**——每条管道的 token 预算、模型路由（廉价模型用于分类，更强模型用于困难案例）、供应商更新时的漂移检测
- **可观测性**——每次 LLM 调用都记录提示词版本、模型、成本、延迟和决策，同时配备标准管道告警，在问题级联之前将其暴露出来
- **核心管道与仓库**——Airflow 编排、从 staging 到 mart 的 dbt 模型、Snowflake 在数十亿行上的性能和成本、AWS 基础设施即代码
- **匹配与去重**——用于跨 13 个市场的公司和个人实体解析的嵌入和检索模式

**我们寻找的**

**必备条件**

- 3+ 年数据工程或数据基础设施经验，拥有你端到端负责的生产管道，以及 2+ 年 Snowflake 经验
- 强大的 Python 和 SQL——生产级、性能意识强、能从容应对超大规模
- 在数据管道中交付过 LLM——生产环境中的提取、丰富或验证，具备结构化输出和标注评估集，能告诉你模型在哪里出错
- 测试框架工程经验——你构建或维护过 LLM 输出的评估或测试框架，并能讲述它们捕获了什么
- 对规则 vs. LLM 有敏锐判断力——你首先想到的是正则表达式或 dbt 测试，并且能为任一选择辩护
- 生产环境 dbt 和 Airflow——模块化、经过测试的模型；能优雅恢复的 DAG
- 云仓库经验——Snowflake 优先；模式设计、查询优化、成本管理
- AI 编码工具是你的工作方式——Claude Code、Cursor 或同等工具，每天使用，并有实际交付成果
- 所有权和系统思维——在更改任何东西之前，你都会权衡上游依赖和下游影响

**高度看重**

- 评估框架和 LLM 追踪（Logfire、OpenTelemetry）
- 嵌入、向量搜索或模糊匹配，用于大规模实体解析
- 微调或蒸馏小型模型以替代昂贵的 LLM 调用
- 大规模 AWS（S3、Lambda、Glue、ECS、RDS）和 PostgreSQL
- Spark 或 PySpark；流处理（Kafka、Kinesis、Snowpipe Streaming）
- 数据

⚑ 岗位信息收录自公开招聘页（Himalayas），以雇主官网为准