发布会·中文精读

直播:马斯克 AI Day 揭晓 Optimus

WATCH: Elon Musk's AI Day With Optimus Reveal - LIVE

中文精读由 AI 基于原始视频与文稿改写,非官方翻译,可能存在偏差或错误;表述以原视频为准。说明

去年特斯拉 AI Day 上那台“机器人”,里面是个穿紧身衣的真人。今年马斯克自己把这件事翻出来当笑话讲:去年那是个穿机器人套装的人,今年我们走了很长的路。他没夸张。新机器人上台时,没有任何缆绳、支架、机械备份,自己走出来的。马斯克说,这是它第一次脱离束缚运行,“我们只是不想让它脸着地”。

这场发布会信息密度极高,前半段讲机器人,后半段讲自动驾驶、芯片、超算和数据管线。工程师一个接一个上台,幻灯片密密麻麻。马斯克把全公司最硬核的技术摊在桌面上,目的写在明面上:说服全世界最聪明的那批人来特斯拉上班。

先把“经济”这个词拆开

马斯克在机器人出场之前,花了几分钟讲一件听起来跟硬件无关的事:经济到底是什么。

他的定义很物理:一个经济体的产出,等于参与生产的实体数量,乘以人均产出效率。 这个公式里,人口一直是个硬约束。地球上能干活的人就那么多,所以经济总量有个天花板。

然后他推了一步:如果人形机器人可以大规模制造,而且成本足够低,那么“参与生产的实体数量”这个变量就失去了上限。公式里被卡住的那一项松开了,经济体的概念本身就模糊了。他的原话是,到那个阶段,经济会变成“准无限”。

这句话听起来像科幻。但紧接着他给了数字:Optimus 的目标成本低于两万美元,低于一辆车。量产规模最终可能是几百万台。他还给了一个量级对比:自动驾驶能把运输的生产效率提升半个数量级到一个数量级,而人形机器人对经济产出的潜在提升是两个数量级。

两个数量级是什么概念。一辆车卖三万美元,一个人形机器人卖两万美元,但车只能运输,机器人能搬运、能装配、能拧螺丝、能去超市。从资产回报的角度看,这两件事不在一个维度上。

他为什么反复强调股权结构

整场演讲里,马斯克有三次主动把话题拐到特斯拉的公司治理上,这在技术发布会里很不寻常。

他的逻辑是这样的:如果特斯拉真的在朝通用人工智能(AGI)的方向走,那么由谁来控制这家公司就变成一个公共安全问题。特斯拉只有一类股票,是上市公司,公众可以买股票、可以投票。他说得很直白:“如果我疯了,你们可以把我开除。”

他补了一句:也许我已经疯了,我不知道。

这话有玩笑成分,但背后的担心是真的。一个掌握几百万台人形机器人、几百万辆自动驾驶汽车、以及全球最大真实世界数据流的公司,如果控制权集中在一个人或一个封闭小圈子里,风险会变得不对称。他给出的解法是把控制权分散到公开市场,让公众能通过买入和投票施加影响。

这个论证站不站得住可以讨论。一家上市公司由持股比例决定话语权,这一点他没有回避。但他至少把问题摆上了台面:做 AGI 的实体,必须能被公众影响。

机器人长得像人,是因为世界是按人设计的

为什么非得是人形?四条腿的机器狗不是更稳吗?

特斯拉硬件团队给出的理由不玄乎:我们的工厂、楼梯、门把手、螺丝刀、货架高度,全部是按人体工程学设计的。 一个能抓握、能拧、能搬的人形机器人,进入现成环境不需要改造世界。反过来,如果换个形态,每进一个场景都要重新设计一遍工具和通道,成本会失控。

手的部分讲得最细。五指、对生拇指、金属腱驱动、六个执行器、十一个自由度。人类手指每秒能移动 300 度,手上有数万个触觉传感器。特斯拉的手做不到这个水平,但做到了两件关键的事:一是大开口的抓握力,二是小物件精细捏取。手指驱动用了不可逆的离合机构,意思是机器人握住东西移动时,手部电机可以关掉,靠机械结构保持握力。

为什么用腱而不用连杆?现场有人问了这个。工程师的回答是:腱可以用金属编织线,强度比合成材料高;零件数量少,量产时更容易;腱驱动还能做反向间隙消除,手指动作不会有顿挫感。配合弹簧,手指收紧靠电机,张开靠弹簧回弹,省掉一半执行器。

膝盖那部分更像一篇论文。人类膝盖算不上简单的铰链,弯曲时力矩是非线性的。特斯拉用四连杆结构去逼近这条非线性曲线,结果是执行器需要的峰值力大幅下降,能耗跟着下降。工程师在台上说了一句挺有意思的话:我们人类就是一袋湿乎乎的果冻和骨头,站立时腿的振动频率根本不在 10 赫兹,所以机器人也没必要追求那么高的刚度,把模态数据喂给控制系统,让它适应就行。

全身 28 个自由度,六种执行器(三种旋转、三种线性),一款线性执行器在测试里举起了一架半吨重的九英尺三角钢琴。躯干中间是 2.3 千瓦时的电池包,够干一整天活。计算单元也在躯干里,头部没有。

车轮上的机器人和腿上的机器人,共用一个大脑

自动驾驶部分由 Ashok 开场。他给了一组对比:去年同期,跑 FSD 测试版的车大约 2,000 辆;今年这个数字是 16 万。过去一年训练了 75,000 个神经网络模型,平均每八分钟出一个,最后只有 281 个真正上线提升了车辆表现。

支撑这一切的是三台自建超算,14,000 张 GPU,其中 10,000 张用于训练,4,000 张用于自动标注。30 PB 的分布式视频缓存,每天有 50 万段视频流进流出,每秒追踪 40 万次视频实例。今年训练基础设施扩容了 40% 到 50%。

核心模型叫 Occupancy Network(占据网络)。八个摄像头的视频流进去,输出一个统一的三维体积表示,对车身周围每一个三维位置预测“这里有没有东西被占据”。它不先识别“这是什么物体”,直接回答“这块空间能不能过”。所以被遮挡的障碍物,它也能瞬间推出来。

有个细节值得记下来:喂给网络的是摄像头原始的 12 位图像,并非常见的 8 位 RGB。多出来的四位让它有 16 倍动态范围,同时因为跳过了 ISP 图像处理环节,延迟更低。这个网络在车上每 10 毫秒跑一次。

然后是车道线。这部分是整场设计里最出乎意料的一处。

早期做法是在图像上做语义分割,把车道线涂出来。这条路在高速上够用,一到路口就崩。路口里的车道是交叉、分叉、互相连通的图结构,你没法用像素涂出来,更没法处理“这里到底是两条车道还是三条车道看不清楚”的模糊情况。

特斯拉的解法是把车道线预测变成一个语言模型问题。车道点变成词,点的位置是词的内容,点与点之间的连接关系变成语法。模型自回归地一个词一个词往外吐:先预测第一个车道点的位置,再预测下一段是延续还是新起点,再预测分叉点回溯到哪个已生成的点,最后吐出一个“句子结束”标记。

John 在台上说得很清楚,他们这么干是因为简单方法真的试过,不行。分割模型遇到不确定的场景会把两条可能的车道都画出来,后处理直接傻掉。而语言模型天然处理多模态分布:它必须选一条路走到底,生成一个连贯的预测,不会同时输出所有可能性。

这个思路还被直接搬到了机器人上。工厂里的“请去厨房”“请到三号货架”,本质上就是预测一组路径和连通关系,跟车道图是同一个数学结构。

Dojo:一家车厂为什么要自己造超算

Pete Bannon 上台第一句话就回答了所有人的疑问:为什么一家汽车公司要造超级计算机。

他的答案是,特斯拉本质上是一家硬核技术公司。自动驾驶团队训练的一些大模型要跑一个多月,这个周期让快速试错变得不可能。目标是 30 倍加速,同时成本和能耗要有竞争力。

设计过程中的关键抉择是:放弃 DRAM,全部用片上 SRAM。 这个决定带来一连串后果。SRAM 容量小但带宽极高、延迟极低,能喂饱运算单元。没有虚拟内存,因为没地方放页表。没有中断,整个加速器是一个裸金属硬件,编译器负责确定性地调度一切。训练范式选了模型并行,因为数据并行要额外的内存容量,他们拿不出来。

Dojo 训练瓦片集成了 25 颗芯片。整机柜的一层托盘装六个瓦片,54 petaflops 算力,640 GB 高带宽内存。一个 ExaPOD 装两套完整加速器,合计 1 exaflop 的机器学习算力。

Rajiv 给了一组硬碰硬的对比。跨 25 颗 Dojo 芯片做批归一化(BatchNorm)的全局归约和广播,8750 个节点协同,耗时 5 微秒。同样的操作在 24 张 GPU 上是 150 微秒。差了两个数量级。

ResNet-50 这种标准基准上,Dojo 已经能做到芯片对芯片打平,而且每颗芯片的批大小只要 8。真正的差距在特斯拉自己的负载上:自动标注网络用生产硬件是 A100 的两倍吞吐,编译器优化后能到三倍以上;占据网络接近三倍。

最有杀伤力的一句话是:一个 Dojo 瓦片可以顶替六个 GPU 机箱,而它的成本低于其中一个 GPU 机箱。

部署上,现在自动标注占用了 4,000 张 GPU、72 个 GPU 机架。同样的吞吐量,四个 Dojo 机柜就够了。第一个 ExaPOD 计划 2023 年第一季度建成,能把特斯拉的自动标注产能翻一倍以上。帕洛阿尔托一共规划了七个 ExaPOD。

中间还讲了一个翻车现场。理论上很美的系统,实测时性能只有预期的 4%。原因是 Dojo 主机的机器学习算力是 GPU 主机的十倍,单台主机上的数据加载器根本喂不饱。他们的解法是自研网络接口卡,把加载主机通过以太网接到 Dojo 网格上,用硬件加速的 DMA 传输。改完之后,占用率从 4% 跳到 97%。

这段插曲比任何参数都有说服力。它说明这套系统真的在跑,也真的踩过坑。

数据引擎:13,900 个片段治好了“路边的车”

Kate 讲了一个具体案例,值得完整复述。

场景是这样的:车辆接近一个路口准备转弯,前方停着一辆车。自动驾驶系统判断这辆车在等红灯,于是减速。实际情况是,那辆车里没人,只是随便停在路边。

特斯拉的诊断工具把这个误判识别出来,修正标签,把片段归入评测集。这类“转弯处的临时停车”案例,一共诊断出 126 个。

然后是从车队里挖数据。用触发器找出所有类似场景,修正标签,最后修了 13,900 个片段加进训练集。模型架构一点没改,只喂了新数据,这个误判就消失了。屏幕上,原来标成橙色的“停车等待”变成了红色的“临时停放”。

她说了句很关键的话:学术界常常把数据当作固定常量,只改模型;特斯拉反过来,数据是最确定、最有效的杠杆。 光是“车辆运动”这一个信号,今年就更新了五次,每次都靠新数据把准确率往上推。

这套流程能跑起来,靠的是三样东西:车队规模、自建的数据管线、以及自动标注能力。

自动标注那部分数字很夸张。为了训练车道网络,需要几千万趟行程的数据,覆盖上百万个路口。人工标注走不通。他们做了多趟重建的自动标注系统:先用多摄像头视觉惯性里程计恢复高精度轨迹和场景结构(车内漂移率每米 1.3 厘米、每米 0.45 毫弧度),再把不同车辆、不同时间经过同一地点的多趟行程做联合对齐和优化,最后由人工做一次确认。

结果:原本需要 500 万小时人工标注的一万趟行程,在集群上 12 小时跑完。 新行程的自动标注每趟只要 30 分钟。Ashok 补充说,后端管线每天要执行 2,000 万个这样的函数调用,流式处理 50 万段视频。他说这套东西像一个生产标签的工厂,有产线、有质量、有库存。

仿真:两周的活,五分钟干完

David 讲仿真。旧金山 Market Street 一个复杂路口,过去美术团队要做两周。现在用自动标注产出的真值数据,加上 Houdini 里的程序化生成工具,五分钟出一个场景。

流程是自动的:道路边界标签生成路面网格,车道图标签重新拓扑,车道线数据扫掠出标线贴花,中央隔离带生成绿化,模块化建筑生成器生成遮挡物,地图数据决定红绿灯和路牌位置,车道邻接关系决定随机车流。

更关键的是,因为一切都是数据驱动的,可以在真值上做扰动。想换材质、换树种、换季节、换成都市场景还是乡村场景,点一下按钮。想让网络更关注某个路口的导向标线,直接在仿真里修改车道图,造出一种现实世界里根本不存在的交通流。

工程上他们做了瓦片化:150 米见方的地理散列瓦片,导出成几何和实例文件,跟渲染引擎解耦。加载器按需召唤任意数量的瓦片,转成虚幻引擎资产。旧金山大部分街道,一个人两周就生成完了。所有这些瓦片可以过夜重跑,保证新加入的语义标签能同步到全部场景。

问答环节里的真话

后半段一个多小时的提问,马斯克回答得比演讲部分更放松,也更容易看出他的判断。

关于什么时候普通人能买到 Optimus,他说:“三年之内,不会超过五年,你大概就能收到一台。” 部署路径是从工厂里最简单的活开始,比如把零件从一个地方搬到另一个地方,或者把零件送进焊接机器人单元。然后有用场景的数量会指数增长。

关于 AGI,他反复强调特斯拉没有专门去做这件事,但 AGI 可能是一个涌现产物。理由是数据量:几百万辆自动驾驶汽车加几百万台人形机器人,带着摄像头、麦克风和其它传感器在真实世界里跑,这个数据集体量远超互联网上能抓到的任何东西。他的原话是,互联网做不到的事情是,它没法拥有几亿个在真实世界里移动的摄像头。

关于 Dojo 会不会对外卖,他的判断是:Dojo 更适合以亚马逊 AWS 那种方式运营,当作产品卖给某家公司并不现实。 这台机器耗电巨大、需要专门冷却,装进别人机房办不到。做成在线服务,让客户用更快的速度、更低的成本训练模型,才是合理的商业模式。他还顺带提了一句“软件 2.0”,意思是未来所有软件都会被神经网络重写一遍,那么训练算力就是新的水电煤。

关于 AI 监管,他的立场很明确:应该有一个政府层面的 AI 监管机构,就像航空、汽车、食品药品都有监管机构一样,因为这些领域都影响公共安全。他补了一句,这件事各国政府目前还没搞明白。

关于 FSD 落地范围,他说技术上年底前具备全球推测试版的能力,但很多国家卡在监管审批。他提到下个月会有一个大幅改进的版本,特别擅长判断快速横向来车的速度。车队内部已经在高速场景跑统一的技术栈,比现有的量产版本好很多,目标是年底前把停车场场景也并进来,实现从停车场到停车场的全程。

关于核心指标,他给了一个非常朴素的答案:每两次必要干预之间能跑多少英里。 这个数字每周都在测,在激进地改善。

有个问题问他,如果回到二十多岁会给自己什么建议。他想了很久,说了一句不太马斯克的话:在夸贾林环礁开发猎鹰一号火箭的那段时间,那座岛很漂亮,但他一次都没去沙滩上喝过一杯酒。“我应该去沙滩上喝一杯的。”

还有一个问题问特斯拉凭什么能同时做这么多事。他的回答里有个词很锋利,叫“蜜罐公司”(Honey Trap)。他说硅谷有一些地方,对工程师看起来不坏,薪水高、环境舒服,但一个优秀工程师走进去,实际产出非常低。这种地方就是蜜罐。特斯拉不干这种事,要求高,你会做成很多事,过程不轻松。

这场发布会真正在说什么

把两小时的内容压缩成一句话:特斯拉在试图证明,真实世界的人工智能需要的是一整套完整的闭环,光有更聪明的算法远远不够。

闭环的每一环都得自己造。车上要有算力,于是有了 FSD 芯片。训练要有算力,于是有了 Dojo。数据要有人采,于是有了几百万辆车的车队。数据要有人标,于是有了多趟重建的自动标注。标完要能造长尾场景,于是有了程序化仿真。模型要能验证,于是有了评测集和数据引擎。机器人要能动,于是有了自研执行器、电池包、手和膝盖。

任何一环外包出去,整个迭代速度就会被最慢的那一环拖住。这才是“垂直整合”在这个语境下的真实含义。它指向的是迭代周期,跟财务报表没什么关系。

至于 Optimus 本身,坦率说,舞台上那台机器人的动作还很初级。它挥手,它展示关节,它被工作人员扶着转了个身。马斯克自己承认,很多技术方案可能选错了路,架构会随时间大幅演进。

但他给的时间表很具体:三年到五年。他给的成本很具体:低于两万美元。他给的产量很具体:最终几百万台。这些数字能不能兑现另说,至少它们可以被验证,可以被证伪。这比“改变世界”这种话有用得多。

给读者的几件事

  • 如果你在做机器学习系统,重点看 Occupancy Network 的设计取舍:它跳过了目标检测,直接预测三维空间占据概率。不问“这是什么”,只回答“能不能过”,反而让遮挡推理变得简单。这个思路可以迁移到很多机器人感知任务上。
  • 把车道线预测做成语言模型这件事值得抄。当一个视觉问题存在多模态歧义、又需要输出结构化的连通图时,自回归的序列生成比稠密分割加后处理更稳。工程上别害怕从别的领域搬现成范式。
  • 数据引擎那套方法论可以直接用:找出模型的系统性误判,用触发器从真实数据里挖同类样本,修正标签,只把最有价值的样本加进训练集,架构保持不动。13,900 个片段解决一个具体的产品问题,这个投入产出比比重新设计网络高得多。
  • 如果你关注机器人创业,注意马斯克对手和膝盖的执念。腱驱动加弹簧回弹省掉一半执行器,四连杆结构把膝盖力矩曲线线性化从而降低峰值力。这些设计的出发点都是量产成本。
  • 如果你想进这个方向,马斯克在台上重复了至少五遍的话是:来特斯拉。他把整场发布会当成招聘广告来做。你可以不认同他的时间表,但这场发布会的技术密度值得你花两小时把原始视频看完,别只看剪辑片段。
YouTube 平台字幕(自动或人工),可能有识别错误,人名与数字请以视频为准。已按语句合并成段落,并去掉 um、uh 等口头填充词。 文稿来源:YouTube(自动/人工字幕)。

0:03foreign welcome everybody let's give everyone a moment to get back in the audience and all right great welcome to Tesla AI day 2022. [Applause] we've got some really exciting things to show you I think you'll be pretty impressed I do want to set some expectations with respect to our Optimus robot

1:08as as you know last year it was just a person in a robot suit but We've now we've come a long way and it's I think you know compared to that it's going to be very impressive and we're going to talk about the advancements in AI for full self-driving as well as how they apply to more generally to real world AI problems like a humanoid robot and and even going beyond that I think there's some potential that what we're doing here at Tesla could make a meaningful contribution to AGI

1:47and and I think actually tells us a good entity to do it from a governance standpoint because we're a publicly traded company we have one class of of stock and that means that the public controls Tesla and I think that's actually a good thing so if I go crazy you can fire me this is important maybe I've gone crazy I don't know so yeah so we're going to talk a lot about our progress in AI autopilot as well as a progress in with with dojo and then we're going to bring the team out and do a long q a so you can ask tough questions whatever you'd like existential questions technical questions we want to have as much time for Q a as possible

2:37so let's see with that because with daily hey guys I'm Milan I work on autopilot and the Tesla and I'm Lizzy a mechanical engineer on the project as well okay so should we should we bring out the bot before we [Applause] four weeks robot without any backup support cranes mechanical mechanisms no cables nothing yeah I want to join with you guys tonight but it was the first time let's see you ready let's go [Applause]

3:23foreign [Music] foreign [Music] self-driving computer that runs in your Tesla cars by the way this is the it's literally the first time the robot has operated without a tether was on stage tonight [Music]

5:06foreign so the robot can actually do a lot more than we just showed you we just didn't want it to fall on its face so we'll we'll we'll show you some videos now of the robot doing a bunch of other things yeah which are less risky yeah we should close the screen guys [Music] yeah we wanted to show a little bit more what we've done over the past few months with apart and just walking around and dancing on stage just humble beginnings but you can see the autopilot neural networks running as is just retrained for the bud directly on that on that new platform that's my watering can yeah when you see a rendered view that's that's the robot what's the that's the world the robot

6:03sees so it's it's very clearly identifying objects like this is the object it should pick up picking it up yeah [Music] we use the same process as we did for autopilot to collect data in train your networks that we then Deploy on the robot that's an example that illustrates the upper body a little bit more

6:30something that we'll like try to nail down in a few months over the next few months I would say to perfection this is really an a station in the Fremont Factory as well that it's working at yep so [Applause] and that's not the only thing we have to show today right yeah absolutely so that what you saw was what we call Bumble C that's our sort of rough development robot using semi-off-the-shelf actuators but we actually have it's gone a step further than that already the team's done an incredible job and we actually have an optimist bot with a fully Tesla designed at both actuators battery pack control system everything it it wasn't quite ready to walk

7:27but I think it will walk in a few weeks but we wanted to show you the robot the the something that's actually fairly close to what will go into production and and show you all the things it can do so let's bring it out do it [Music]

8:05all right [Music] thank you foreign [Applause] that we expect to have in Optimus production unit one which is the ability to move all the fingers independently move the to have the thumb have two degrees of freedom so it has opposable thumbs and both left and right hand so it's able to operate tools and do useful things our goal is to make a a useful humanoid robot as quickly as possible and we've also designed it using the same discipline that we use in designing the car which is to say to to design it for manufacturing such that it's possible to make the robot at in high volume at low cost with high reliability so that's incredibly important I mean

9:22you've all seen very impressive humanoid robot demonstrations and that that's great but what are they missing they're missing a brain they don't have the intelligence to navigate the world by themselves and they're they're also very expensive and made in low volume whereas this this is the optimistic society and extremely capable robot but made in very high volume probably ultimately millions of units and it is expected to cost much less than a car I would say probably less than twenty thousand dollars would be my guess [Applause]

10:07the the potential for optimistic is I think appreciated by very effective people hey as usual Tesla demos are coming in hot so okay that's good that's good yeah the teams put on put in and the team has put in an incredible amount of work it's working days you know seven days a week running the 3am oil that to to get to the demonstration today I'm super proud of what they've done is they've really done done a great job I just like to give a hand to the whole option of this team

10:52[Applause] so you know that now there's still a lot of work to be done to refine Optimus and improve it obviously this is just Optimus version one and that's really why we're holding this event which is to convince some of the most talented people in the world like you guys to join Tesla and help make it a

11:18reality and bring it to fruition at scale such that it can help millions of people and the the and the potential like I said is is really boggles the mind because you have to say like what what is an economy an economy is sort of productive entities times the productivity Capital times output productivity per capita at the point in which there is not a limitation on capita the it's not clear what an economy even means at that point an economy becomes quasially infinite so what what you know taken to fruition in the hopefully benign scenario the this means a future of

12:06abundance a future where there is no poverty where people you can have whatever you want in terms of products and services it really is a a fundamental transformation of civilization as we know it obviously we want to make sure that transformation is a positive one and safe and but that's also why I think Tesla as an entity doing this being a single class of stock publicly traded owned by the public is very important and should not be overlooked I think this is essential because then if the public doesn't like what Tesla's doing the public can buy shares in Tesla and vote differently this is a big deal like it's very important that that I can't just do what I want

13:04you know sometimes people think that but it's not true so you know that's it's very important that the the corporate entity that has that that makes this happen is something that the public can properly influence and so I think the Tesla structure is is ideal for that and like I said that you know self-driving cars will certainly have a tremendous impact on the world I think they will improve the productivity of Transport by at least a half order of magnitude perhaps an order of magnitude perhaps more

13:50optimists I think has maybe a two order of magnitude potential Improvement in economic output like like it's not clear it's not clear what the limit actually even is so but we need to do this in the right way we need to do it carefully and safely and ensure that the outcome is one that is beneficial to civilization and and one that Humanity once I can't this is also it's extremely important obviously so and I hope you will consider joining Tesla to achieve those goals

14:42and Tesla we really care about doing the right thing here always aspire to do the right thing and and really not pave the road to hell with good intentions and I think the road is Roadhouse mostly behaved with bad intentions but every now and again there's a good intention in there so we want to do the right thing so you know consider joining us and helping make it happen with that let's let's move on to the next phase right on thank you Elon

15:14all right so you've seen a couple robots today let's do a quick timeline recap so last year we unveiled the Tesla bot concept but a concept doesn't get us very far we knew we needed a real development and integration platform to get real-life learnings as quickly as possible so that robot that came out and did the little routine for you guys we had that within six months built working on software integration Hardware upgrades over the months since then but in parallel we've also been designing the Next Generation this one over here so this guy is rooted in the the foundation of sort of the vehicle design process you know we're leveraging all of

15:52those learnings that we already have obviously there's a lot that's changed since last year but there's a few things that are still the same you'll notice we still have this really detailed focus on the true human form we think that matters for a few reasons but it's fun we spend a lot of time thinking about how amazing the human body is we have this incredible range of motion typically really amazing strength a fun exercise is if you put your fingertip on the chair in front of you you'll notice that there's a huge range of motion that you have in your shoulder and your elbow for example without moving your fingertip you can move those joints all over the place

16:30but the robot you know its main function is to do real useful work and it maybe doesn't necessarily need all of those degrees of freedom right away so we've stripped it down to a minimum sort of 28 fundamental degrees of freedom and then of course our hands in addition to that humans are also pretty efficient at some things and not so efficient in other times so for example we can eat a small amount of food to sustain ourselves for several hours that's great but when we're just kind of sitting around no offense but we're kind of inefficient we're just sort of burning energy so on the robot platform what we're going to do is we're going to minimize

17:06that idle power consumption drop it as low as possible and that way we can just flip a switch and immediately the robot turns into something that does useful work so let's talk about this latest generation in some detail shall we so on the screen here you'll see in Orange are actuators which we'll get to in a little bit and in blue our electrical system so now that we have our sort of human-based research and we have our first development platform we have both research and execution to draw from for this design again we're using that vehicle design foundation so we're taking it from concept through design and Analysis and then build and validation

17:47along the way we're going to optimize for things like cost and efficiency because those are critical metrics to take this product to scale eventually how are we going to do that well we're going to reduce our part count and our power consumption of every element possible we're going to do things like reduce the sensing and the wiring at our extremities you can imagine a lot of mass in your hands and feet is going to be quite difficult and power consumptive to move around and we're going to centralize both our power distribution and our compute to the physical center of the platform so in the middle of our torso actually it is the Torso we have our battery pack

18:23this is sized at 2.3 kilowatt hours which is perfect for about a full day's worth of work what's really unique about this battery pack is it has all of the battery Electronics integrated into a single PCB within the pack so that means everything from sensing to fusing charge management and power distribution is all on one all in one place we're also leveraging both our Vehicle Products and our Energy Products to roll all of those key features into this battery so that's streamlined manufacturing really efficient and simple cooling methods battery management and also safety and of course we can leverage Tesla's existing infrastructure and supply chain

19:07to make it so going on to sort of our brain it's not in the head but it's pretty close also in our torso we have our Central Computer so as you know Tesla already ships full self-driving computers in every vehicle we produce we want to leverage both the autopilot hardware and the software for the humanoid platform but because it's different in requirements and in form factor we're going to change a few things first so we still are gonna it's gonna do everything that a human brain does processing Vision data making Split Second decisions based on multiple sensory inputs and also Communications so to support Communications it's equipped with wireless connectivity as

19:50well as audio support and then it also has Hardware level security features which are important to protect both the robot and the people around the robot so now that we have our sort of core we're going to need some limbs on this guy and we'd love to show you a little bit about our actuators and our fully functional hands as well but before we do that I'd like to introduce Malcolm who's going to speak a little bit about our structural foundation for the robot

20:19[Applause] thank you Vicky Tesla have the capabilities to analyze highly complex systems don't get much more complex than a crash you can see here a simulated crash and model 3 superimposed on top of the actual physical crash it's actually incredible how how accurate it is just to give you an idea of the complexity of this model it includes every nut bottle washer every spot Weld and it has 35 million degrees of freedom quite amazing and it's true to say that if we didn't have models like this we wouldn't be able to make the safest cars in the world so can we utilize our capabilities and our methods from the automotive side to influence a robot

21:08well we can make a model and since we had crash software we used the same software here we can make it fall down the purpose of this is to make sure that if it falls down ideally it doesn't but it's superficial damage we don't want to for example break its gearbox at its arms that's equivalent of a dislocated shoulder of a robot difficult and expensive to fix so we wanted to dust itself off get on with a job that's been given if we'd also take the same model and we can drive the actuators using the input from a previously solved model bringing it to life so this is producing the Motions for the tasks we want the robot to do these tasks are picking up boxes turning

21:52squatting walking upstairs whatever the set of tasks are we can play to the model this is showing just simple walking we can create the stresses in all the components that helps us to optimize the components these are not dancing robots these are actually the modal Behavior the first five modes of the robot and typically when people make robots they make sure the first mode is up around the top single figures up towards 10 Hertz who is to do this is to make the controls of walking easier it's very difficult to walk if you can't guarantee where your foot wobbling around that's okay to make one robot we want to make thousands maybe Millions we haven't

22:33got the luxury of making them from carbon fiber titanium we want to make them on plastic things are not quite so stiff so we can't have these high targets I'll call them dumb targets we've got to make them work at lower targets so is that is that going to work well if you think about it sorry about this but we're just bags of soggy jelly and Bones thrown in we're not high frequency if I stand on my leg I don't vibrate at 10 Hertz we people operate at a low frequency so we know the robot actually can it just makes controls harder so we take the information from this the modal data and the stiffness and feed that into the control system that allows it to walk

23:16just changing tax slightly looking at the knee we could take some inspiration from biology and we can look to see what the mechanical advantages of the knee is it turns out it actually represents quite similar to four bar link and that's quite non-linear that's not surprising really because if you think when you bend your leg down the torque on your knee is much more when it's bent than it is when it's straight so you'd expect a non-linear function and in fact the biology is non-linear this matches it quite accurately so that's a representation the four by link is obviously not physically four by link as I said the characteristics are similar but me betting down that's not

23:56very scientific let's be a bit more scientific we've played all the tasks through the through this graph but this is showing pictures of walking squatting the tasks I said we did on the stress and that's the the talk at scene at the knee against the knee bend on the horizontal axis this is showing the requirement for the need to do all these tasks and then put a curve through it surfing over the top of the piece and that's saying this is what's required to make the robot do these tasks

24:30so if we look at the four bar link that's actually the green curve and it's saying that the non-linearity of the four by link is actually linearized the characteristic of the force what that really says is that's lowered the force that's what makes the actuator have the lowest possible Force which is the most efficient we want to burn energy up slowly what's the blue curve well the blue curve is actually if we didn't have a four bar link we just had an arm sticking out of my leg here with a with an actuator on it a simple two bar link that's the best we could do with a simple two bar link and it shows that that would create much more force in the

25:04actuator which would not be efficient so what's that look like in practice well let's just see but it's very tightly packaged in the knee you'll see good transparent in a second you'll see the full bar link there it's operating on the actuator this is determined the force and the displacements on the actuator and now pass you over to constantina to tell you a lot more detail about how these actuators are made and designed optimized thank you thank you Michael [Applause]

25:39so I am I would like to talk to you about the design process and the actuator portfolio in our robot so there are many similarities between a car and the robot when it comes to powertrain design the the most important thing that matters here is energy mass and cost we are carrying over most of our designing experience from the car to the robot so in the particular case you see a car with two drive units and the drive units are used in order to accelerate the car 0 to 60 miles per hour time or drive a City Drive side while the robot that has 28 actuators and it's not obvious what are the tasks at the actuator level so we have tasks that are higher level like walking or

26:34climbing stairs or carrying a heavy object which need to be translated into joint into joint specs therefore we use our model that generates the torque speed trajectories for our joints which subsequently is going to be fed in our optimization model and to run through the optimization process this is one of the scenarios that the robot is capable of doing which is turning and walking so when we have this torque speed trajectory we laid over an efficiency map of an actuator and we are able along the trajectory to generate the power

27:18consumption and the energy accumulative energy for the task versus time so this allows us to define the system cost for the particular actuator and put a simple Point into the cloud then we do this for hundreds of thousands of actuators by solving in our cluster and the red line denotes the Pareto front which is the preferred area where we will look for optimal so the X denotes the preferred actuator design we have picked for this particular joint so now we need to do this for every joint we have 28 joints to optimize and we parse our cloud we parse our Cloud again for every joint spec and the red axis this time denote the bespoke actuator designs for every

28:05joint the problem here is that we have too many unique actuator designs and even if we take advantage of the Symmetry still there are too many in order to make something Mass manufacturable we need to be able to reduce the amount of unique actuator designs therefore we run something called commonality study which we parse our Cloud again looking this time for actuators that simultaneously Meet The Joint performance requirements for more than one joint at the same time so the resulting portfolio is six actuators and they show in a color map the middle figure and the actuators can be also viewed in this slide we have three rotary and three linear actuators all of which have a

28:52great output force or Torque per Mass the rotary actuator in particular has a mechanical clutch integrated on the high speed side angular contact ball bearing and on the high speed side and on a low speed side a cross roller bearing and the gear train is a strain wave gear and there are three integrated sensors here and the bespoke permanent magnet machine the linear actuator

29:30I'm sorry the linear actuator has planetary rollers and an inverted planetary Screw As a gear train which allows efficiency and compaction and durability so in order to demonstrate the force capability of our linear actuators we have set up an experiments in order to test it under its limits

29:57and I will let you enjoy the video so our actuator is able to lift a half tone nine foot concert grand piano and this is a requirement it's not something nice to have because our muscles can do the same when they are direct driven when they are directly driven or quadricep muscles can do the same thing it's just that the knee is an up gearing linkage system that converts the force into velocity at the end effector of our heels for purposes of giving to the human body agility so this is one of the main things that are amazing about the human body and I'm concluding my part at this point and I would like to welcome my colleague Mike who's going to talk to you about hand

31:04design thank you very much thanks constantinos so we just saw how powerful a human and a humanoid actuator can be however humans are also incredibly dexterous human hand has the ability to move at 300 degrees per second there's tens of thousands of tactile sensors and it has the ability to grasp and manipulate almost every object in our daily lives for our robotic hand design we were inspired by biology we have five fingers an opposable thumb our fingers are driven by metallic tendons that are both flexible and strong we have the ability to complete wide aperture power grasps while also being optimized for precision gripping of small thin and delicate objects

31:55so why a human-like robotic hand well the main reason is that our factories and the world around us is designed to be ergonomic so what that means is that it ensures that objects in our Factory are graspable but it also ensures that new objects that we may have never seen before can be grasped by the human hand and by our robotic hand as well the converse there is is pretty interesting because it's saying that these objects are designed to our hand instead of having to make changes to our hand to accompany a new object some basic stats about our hand is that has six actuators and 11 degrees of freedom it has an in-hand controller which drives the fingers and receives sensor

32:34feedback sensor feedback is really important to learn a little bit more about the objects that we're grasping and also for proprioception and that's the ability for us to recognize where our hand is in space one of the important aspects of our hand is that it's adaptive this adaptability is involved essentially as complex mechanisms that allow the hand to adapt to the objects that's being grasped another important part is that we have a non-back drivable finger drive this clutching mechanism allows us to hold and transport objects without having to turn on the hand Motors you just heard how we went about going we went about designing the Tesla bot

33:10Hardware now I'll hand it off to Milan and our autonomy team to bring this robot to life thanks Mike all right so all those cool things we've shown earlier in the video were posted possible just in a matter of a few months thanks to the amazing word that we've done autopilot over the past few years most of those components ported quite easily over to the Bots environment if you think about it we're just moving from a robot on Wheels to a robot on legs so some of those components are pretty similar and some other require more heavy lifting so for example our computer vision neural networks reported directly from autopilot to the Bots situation

33:57it's exactly the same occupancy Network that we're talking to a little bit more details later with the other pilot team that is now running on the bot here in this video the only thing that changed really is the training data that we had to recollect we're also trying to find ways to improve those occupancy networks using work made on your Radiance fields to get really great volumetric rendering of the Bots environments for example here some Machinery that the bot might have to interact with another interesting problem to think about is in indoor environments mostly with that sense of GPS signal how do you get about to navigate to its destination

34:39say for instance to find its nearest charging station so we've been training more neural networks to identify high frequency features key points within the Bots camera streams and track them across frames over time as the bot navigates to its its environment and we're using those points to get a better estimate of the Bots pose and trajectory within its environment as it's working thank you we also did quite some work on the simulation side and this is literally the autopilot simulator to which we've integrated the robot's Locomotion code and this is a video of the motion control code running into your product simulator simulator showing the

35:22evolution of the robot's work over time and so as you can see we started quite slowly in April and start accelerating as we unlock more joints and deeper more Advanced Techniques like arms balancing over the past few months and so Locomotion is specifically one component that's very different as we're moving from the car to the Bots environment and so I think it warrants a little bit more depth and I'd like my colleagues to start talking about this now foreign

35:56hi everyone I'm Felix I'm a robotics engineer on the project and I'm going to talk about walking seems easy right people do it every day you don't even have to think about it but there are some aspects of walking which are challenging from engineering perspective for example physical self-awareness that means having a good representation of yourself what is the length of your limbs what is the mass of your limbs what is the size of your feet all that matters also having an energy efficient gate you can imagine there's different styles of walking and all of them are equally efficient most important keep balance don't fall and of course also coordinate the motion

36:41of all of your limbs together so now humans do all of this naturally but as Engineers or roboticists we have to think about these problems and if I'm going to show you how we address them in our Locomotion planning and control stack so we start with Locomotion planning and our representation of the bond that means the model of the robot's kinematics Dynamics and the contact properties and using that model and the desired path for the Bots our Locomotion planner generates reference trajectories for the entire system this means feasible trajectories with respect to the assumptions of our model the planner currently Works in three stages it starts planning footsteps and

37:25ends with the entire motion photo system and let's dive a little bit deeper in how this works so in this video we see footsteps being planned over planning Horizon following the desired path and we start from this and add then for trajectories that connect these footsteps using toe off and yield strike just as the humans just as humans do and this gives us a larger stride and less knee Bend for higher efficiency of the system the last stage is then finding a sense of mass trajectory which gives us a fee dynamically feasible motion of the entire system to keep balance as we all know plans are good but we also have to realize them in reality let's say you know see how we can do

38:09this [Applause] thank you Felix hello everyone my name is Anand and I'm going to talk to you about controls so let's take the motion plan that Felix just talked about and put it in the real world on a real robot let's see what happens takes a couple steps and falls down well that's a little disappointing but we are missing a few key pieces here which will make it work now as Felix mentioned the motion planner is using an idealized version of itself and a version of reality around it this is not exactly correct it also expresses its intention through

39:02trajectories and wrenches wrenches of forces and torques that it wants to exert on the World to locomote reality is way more complex than any similar model also the robot is not simplified it's got vibrations and modes compliance sensor noise and on and on and on so what does that do to the real world when you put the bot in the real world well the unexpected forces cause unmodal Dynamics which essentially the planet doesn't know about and that causes destabilization especially For A system that is dynamically stable like biped locomotion so what can we do about it well we measure reality we use sensors and our understanding of the world to do state

39:50estimation and status to me here you can see the attitude and pelvis pose which is essentially the vestibular system in a human along with the center of mass trajectory being tracked when the robot's walking in the office environment now we have all the pieces we need in order to close the loop so we use our better bot model we use the understanding of reality that we've gained through State estimation and we compare what we want versus what we expect the reality we expect that reality is doing to us in order to add corrections to the behavior of the robot here the robot certainly doesn't appreciate being poked but it does an admirable job of staying upright

40:36the final Point here is a robot that walks is not enough we needed to use its hands and arms to be useful let's talk about manipulation foreign [Applause] my name is Eric robotics engineer on teslabot and I want to talk about how we've made the robot manipulate things in the real world we wanted to manipulate objects while looking as natural as possible and also get there quickly so what we've done is we've broken this process down into two steps first is generating a library of natural motion references or we could call them demonstrations and then we've adapted these motion references online to the current real world situation so let's say we have a human

41:32demonstration of picking up an object we can get a motion capture of that demonstration which is visualized right here as a bunch of keyframes representing the locations of the hands the elbows the Torso we can map that to the robot using inverse kinematics and if we collect a lot of these now we have a library that we can work with but a single demonstration is not generalizable to the variation in the real world for instance this would only work for a box in a very particular location so what we've also done is run these reference trajectories through a trajectory optimization program which solves for where the hand should be how the robot should balance

42:15during when it needs to adapt the motion to the real world so for instance if the box is in this location then our Optimizer will create this trajectory instead next Milan's going to talk about what's next for the Optimus Tesla live thanks thanks Eric right so hopefully by now you guys got a good idea of what we've been up to over the past few months we started doing something that's usable but it's far from being useful there's still a long and exciting road ahead of us I think the first thing within the next few weeks is to get Optimus at least at par with Bumble C the other bug prototype you saw earlier and probably Beyond we're also going to start

43:07focusing on the real use case at one of our factories and really gonna try to try to nail this down and I run out all the elements needed to deploy this product in the real world I was mentioning earlier you know indoor navigation graceful for management or even servicing all components needed to scale this product up but I don't know about you but after seeing what we've shown tonight I'm pretty sure we can get this done within the next few months or years and I make this product a reality and change the entire economy so I would like to thank the entire Optimus team for their hard work over the past few months I think it's pretty amazing all of this was done in barely

43:48six or eight months thank you very much [Applause] hey everyone hi I'm Ashok I lead the autopilot team alongside Milan God is coming so hard to top that Optimus section he'll try nonetheless anyway every Tesla that has been built over the last several years we think has the hardware to make the car drive itself we have been working on the software to add higher and higher levels of autonomy this time around last year we had roughly 2 000 cars driving our FSD beta software since then we have significantly improved the software as robustness and capability that we have now shipped it to 160 000 customers as of today

44:52if [Applause] this is not come for free it came from the sweat and blood of the engineering team over the last one year for example we trained 75 000 neural network models just last one year that's roughly a model every eight minutes that's you know coming out of the team and then we evaluate them on our large clusters and then we ship 281 of those models that actually improve the performance of the car and this space of innovation is happening throughout the stack the the planning software the infrastructure the tools even hiring everything is progressing to the next level the FSG beta software is quite capable of driving the car you should be able to navigate from

45:41parking lot to parking lot handling CDC driving stopping for traffic lights and stop signs negotiating with objects at intersections making turns and so on this comes from the camera streams that go through our neural networks that run on the car itself it's not coming back to the server or anything it runs on the car and produces all the outputs to form the world model around the car and the planning software drives the car based on that today we'll go into a lot of the components that make up the system the occupancy Network acts as the base geometry layer of the system this is a multi-camera video neural network that from the images predicts the full

46:28physical occupancy of the world around the robot so anything that's physically present trees walls buildings cars walls what have you it predicts if we specifically present it predicts them along with their future motion on top of this base level of geometry we have more semantic layers in order to navigate the roadways we need the lanes of course but then the roadways have lots of different lanes and they connect in all kinds of ways so it's actually a really difficult problem for typical computer vision techniques to predict the set of planes and their connectivities so we reached all the way into language Technologies and then pulled the state

47:08of the art from other domains and not just computer vision to make this task possible for vehicles we need their full kinematic state to control for them all of this directly comes from neural networks video streams raw video streams come into the networks go through a lot of processing and then outputs the full kinematic state that positions velocities acceleration jerk all of that directly comes out of networks with minimal post processing that's really fascinating to me because how how is this even possible what world do we live in that this magic is possible that these networks predicts fourth derivatives of these positions when people thought we couldn't even detect

47:48these objects my opinion is that it did not come for free it required tons of data so we had a bit sophisticated Auto labeling systems that Shone through raw sensor data run a ton of offline compute on the servers it can take a few hours run expensive neural networks distill the information into labels that train our in-car neural networks on top of this we also use our simulation system to synthetically create images and since it's a simulation we trivially have all the labels all of this goes through a well-oiled data engine pipeline where we first train a baseline model with some data ship it to the car see what the failures are and once you know the failures

48:36we mine the fleet for the cases where it fails provide the correct labels and add the data to the training set this process systematically fixes the issues and we do this for every task that runs in the car yeah and to train these new massive neural networks this year we expanded our training infrastructure by roughly 40 to 50 percent so that sits us at about 14 000 gpus today across multiple training clusters in the United States we also worked on our AI compiler which now supports new operations needed by those neural networks and map them to the the best of our underlying Hardware resources and our inference engine today is capable of Distributing the execution of

49:20a single neural network across two independent system on ships essentially two independent computers interconnected within the simple self-driving computer and to make this possible we have to keep a tight control on the end-to-end latency of this new system so we deployed more advanced scheduling code across the full FSD platform

49:41all of these neural networks running in the car look at that produce the vector space which is again the model of the world around the robot or the car and then the planning system operates on top of this coming up with trajectories that avoid collisions or smooth make progress towards the destination using a combination of model based optimization plus neural network that helps optimize it to be really fast today we are really excited to present progress on all of these areas we have the engineering leads standing by to come in and explain these various blocks and these power not just the car but the same components also run on the Optimus robot that Milan showed earlier

50:20with that I welcome panel to start talking about the planning section hi all I'm paril Jain let's use this intersection scenario to dive straight into how we do the planning and decision making in autopilot so we are approaching this intersection from a side street and we have to yield to all the crossing vehicles right with as we are about to enter the intersection The Pedestrian on the other side of the intersection decides to cross the road without a crosswalk now we need to yield to this pedestrian yield to the vehicles from the right and also understand the relation between The Pedestrian and the vehicle on the other side of the intersection

51:07so a lot of these intra object dependencies that we need to resolve in a quick glance and humans are really good at this we look at a scene understand all the possible interactions evaluate the most promising ones and generally end up choosing a reasonable one so let's look at a few of these interactions that autopilot system evaluated we could have gone in front of this pedestrian with a very aggressive launch in a lateral profile now obviously we are being a jerk to The Pedestrian and we would spook The Pedestrian and his cute pet we could have moved forward slowly short for a gap between The Pedestrian or and the vehicle from the right again we are being a jerk to the vehicle

51:48coming from the right but you should not outright reject this interaction in case this is only safe interaction available lastly the interaction we ended up choosing stay slow initially find the reasonable Gap and then finish the maneuver after all the agents pass now evaluation of all of these interactions is not trivial especially when you care about modeling the higher order derivatives for other agents for example what is the longitudinal joke required by the vehicle coming from the right when you assert in front of it relying purely on collision checks with marginal predictions will only get you so far because you will miss out on a lot of valid interactions

52:31this basically boils down to solving a multi-agent joint trajectory planning problem over the trajectories of ego and all the other agents now how much ever you optimize there's going to be a limit to how fast you can run this optimization problem it will be close to close to order of 10 milliseconds even after a lot of incremental approximations now for a typical crowded unpredictable lift say you have more than 20 objects each object having multiple different future modes the number of relevant interaction combinations will blow up

53:06we the planner needs to make a decision every 50 milliseconds so how do we solve this in real time we rely on a framework what we call as interaction search which is basically a parallelized research over a bunch of maneuver trajectories the state space here corresponds to the kinematic state of ego the kinematic state of other agents the nominal future multiple multimodal predictions and all the static entities in the scene the action space is where things get interesting we use a set of maneuver trajectory candidates to Branch over a bunch of interaction decisions and also incremental goals for a longer Horizon maneuver Let's Walk Through This research very

53:50quickly to get a sense of how it works we start with a set of vision measurements namely Lanes occupancy moving objects these get represented as past extractions as well as latent features we use this to create a set of goal candidates Lanes again from the lanes Network or unstructured regions which correspond to a probability mask derived from Human demonstrations once we have a bunch of these gold candidates we create seed trajectories using a combination of classical optimization approaches as well as our Network planner again trained on data from the customer Fleet now once we get a bunch of these free trajectories we use them to start branching on the interactions

54:33we find the most critical interaction in our case this would be the interaction with respect to The Pedestrian whether we assert in front of it or yield to it obviously the option on the left is a high penalty option it likely won't get prioritized so we Branch further onto the option on the right and that's where we bring in more and more complex interactions building this optimization problem incrementally with more and more constraints and that research keeps flowing branching on more interactions branching on more goals now a lot of tricks here lie in evaluation of each of each of this node of the research inside each node initially we started with creating

55:12trajectories using classical optimization approaches where the constraints like I described would be added incrementally and this would take close to one to five milliseconds per action now even though this is fairly good number when you want to evaluate more than 100 interactions this does not scale so we ended up building lightweight queryable networks that you can run in the loop of the planet these networks are trained on human demonstrations from the fleet as well as offline solvers with relaxed time limits with this we were able to bring the rundown runtime down to close 200 microseconds per action

55:56now doing this alone is not enough because you still have this massive research that you need to go through and you need to efficiently prune the search space so you need to do a do scoring on each of these trajectories few of these are fairly standard you do a bunch of collision checks you do a bunch of comfort analysis what is the girl can access required for a given maneuver the customer Fleet data plays an important role here again we run two sets of again lightweight variable networks both really augmenting each other one of them trained from interventions from the FST beta Fleet which gives a score on How likely is a given maneuver to result in

56:34interventions over the next few seconds and second which is purely on human demonstrations human driven data giving a score on how close is your given selected action to a human driven trajectory the scoring helps us prune the search space keep branching further on the interactions and focus the compute on the most promising outcomes

56:57the cool part about this architecture is that it allows us to create a cool blend between data driven approaches where you don't have to rely on a lot of hand engineered costs but also ground it in reality with physics-based checks a lot of what what I described was with respect to the agents we could observe in the scene but the same framework extends to objects behind occlusions

57:22use the video feed from 8 cameras to generate the 3D occupancy of the world the blue mask here corresponds to the visibility region we call it it basically gets blocked at the first occlusion you see in the scene we consume this visibility mask to generate what we call as ghost objects which you can see on the top left now if you model the spawn regions and the state transitions of this ghost objects correctly if you tune your control response as a function of the existence likelihood you can extract some really nice human-like behaviors now I'll pass it on on to fill to describe more on how we generate these occupancy Networks

58:09hey guys my name is Phil I will share the details of the occupancy Network we build over the past year this network is our solution to model the physical work in 3D around our cars and it is currently not shown in our customer facing visualization and what we will see here is the raw Network output from our internal Dev tool

58:33the occupancy Network takes video streams of all our 80 cameras as input produces a single unified volumetric occupancy in Vector space directly for every 3D location around our car it predicts the probability of that location being occupied or not since it has video contacts it is capable of predicting obstacles that are occluded instantaneously for each location it also produces a set of semantics such as curb car pedestrian and low debris as color coded here

59:14occupancy flow is also predicted for motion since the model is a generalized Network it does not tell static and dynamic objects explicitly it is able to produce and model the random motions such as the swerving trainer here this network is currently running in all Teslas with FSD computers and it is incredibly efficient runs about every 10 milliseconds with our neural accelerator so how does this work let's take a look at the architecture first we Rectify each camera images with the camera calibration and the images were shown here were given to the network it's actually not the typical 8-bit RGB image as you can see from the first imagery on top we're giving the

1:00:0112-bit raw photo account image to the network since it has four bits more information it has 16 times better dynamic range as well as reduced latency since we don't have the wrong ISP in Adobe anymore we use a set of records and back with FPS as a backbone to extract images space features next we construct a set of 3D position query along with the IMG space features as keys and values fit into an attention module the output of the attention module is high dimensional spatial features these special features are aligned temporarily using vehicle odometry to derive motion last this spatial temporal features go through a set of D convolution to produce the final occupancy and

1:00:53occupancy flow output they're formed as fixed size boxer gray which might not be precise enough for planning on control in order to get a higher resolution we also produce per voxel feature Maps which will feed into MLP with 3D spatial Point queries to get position and semantics at any arbitrary location

1:01:17after knowing the model better let's take a look at another example here we have an articular bus parked on right side row highlighted as a L-shaped boxer here as we approach the bus starts to move the blue the front of the cart turns blue first indicating the model predicts the frontal bus has a down zero occupancy flow and the s-bus keeps moving the entire bus turns blue and you can also see that the network predicts the precise curvature of the bus well this is a very complicated problem for traditional object detection Network as you have to see whether I'm going to use one cuboid or perhaps a two to fit the curvature but for occupation Network

1:02:02since all we care about is the occupancy in the visible space and we'll be able to model the curvature precisely besides the box of gray the occupancy Network also produces a drivable surface the drivable surface has both 3D geometry and semantics they are very useful for control especially on healing and curvy roads the surface and the voxel gray are not predicted independently instead the voxel grid actually aligns with the surface implicitly here we are at a hero Quest where you can see the 3D geometry of the surface being predicted nicely planner can use this information to decide perhaps we need to slow down more for the Hillcrest and as you can also

1:02:51see the voxel grade aligns with the surface consistently besides the Box source and the surface we're also very excited about the recent breakthrough in neural Radiance field on Earth looking into both incorporate some of the lighter features into occupancy Network training as well as using our Network output as input state for Nerf

1:03:16as a matter of fact Ashok is very excited about this this has been his personal weekend project for a while these nerves because I think you know the Academia is building a lot of these Foundation models for language using like tons of large data sets for language but I think for vision nerves are going to provide the foundation models for computer vision because they are grounded in geometry and geometry gives us a nice way to supervise these networks and freezes of the requirement to Define an ontology and the supervision is essentially free because you just have to differentiably render these images so I think in the future this occupancy Network idea where you know

1:03:59images come in and then the network produces a consistent volumetric representation of the scene that can then be differentially rendered into any image that was observed I I personally think is a future of computer vision and you know we do some initial work on it right now but I think in the future both at Tesla and in the Academia we will see that these combination of One-Shot prediction of volumetric occupancy will be that's my personal bet sexual so here's an example early result of a 3D Reconstruction from our free data instead of focusing on getting perfect RGB reprojection in image space our primary goal here is to accurately represent the warnings 3D space for

1:04:46driving and we want to do this for all our free data over the world in all weather and lighting conditions and obviously this is a very challenging problem and we're looking for you guys to help finally the occupancy network is trained with large auto level data set without any human in the loop and with that I'll pass to Tim to talk about what it takes to train this network thanks Phil [Applause] all right hey everyone let's talk about some training infrastructure so we've seen a couple videos you know four or five I think and care more and worry more about a lot more Clips on that so we've been looking at the occupancy networks just from Phil

1:05:34just fills videos it takes 1.4 billion frames to train that Network what you just saw and if you have a hundred thousand gpus it would take one hour but if you have one GPU it would take a hundred thousand hours so that is not a Humane time period that you can wait for your training job to run right we want to ship faster than that so that means you're going to need to go parallel so you need a more compute for that that means you're going to need a supercomputer so this is why we've built in-house three supercomputers comprising of 14 000 gpus where we use 10 000 gpus for training and around four thousand gpus for auto labeling all these videos are stored in 30

1:06:18petabytes of indistributed managed video cache you shouldn't think of our data sets as fixed let's say as you think of your imagenet or something you know with like a million frames you should think of it as a very fluid thing so we've got a half a million of these videos flowing in and out of this cluster these clusters every single day and we track 400 000 of these kind of python video instantiations every second so that is that's a lot of calls we're gonna need to capture that in order to govern the retention policies of this distributed video cache so underlying all of this is a huge amount of infra all of which we build and manage in-house

1:07:02so you cannot just buy you know 40 000 gpus and then a 30 petabytes of Flash nvme and just put it together and let's go train it actually takes a lot of work and I'm gonna go into a little bit of that what you actually typically want to do is you want to take your accelerator so that it could be the GPU or Dojo which we'll talk about later and because that's the most expensive component that's where you want to put your bottleneck and so that means that every single part of your system is going to need to outperform this accelerator and so that is really complicated that means that your storage is going to need to have the size and the bandwidth to deliver all the data

1:07:43down into the nodes these nodes need to have the right amount of CPU and memory capabilities to feed into your machine learning framework this machine learning framework then needs to hand it off to your GPU and then you can start training but then you need to do so across hundreds or thousands of GPU in a reliable way in logstap and in a way that's also fast so you're also going to need an interconnect extremely complicated we'll talk more about dojo in a second so first I want to take you through some optimizations that we've done on our cluster so we're getting in a lot of videos and video is very much unlike let's say training on images or text which I think

1:08:23is very well established video is quite literally a dimension more complicated and so that's why we needed to go end to end from the storage layer down to the accelerator and optimize every single piece of that because we train on the photon count videos that come directly from our Fleet we train on those directly we do not post process those at all the way it's just done is we seek exactly to the frames we select for our batch we load those in including the frames that they depend on so these are your iframes or your keyframes we package those up move them into shared memory move them into a double bar from the GPU and then use the hardware

1:09:02decoder that's only accelerated to actually decode the video so we do that on the GPU natively and this is all in a very nice python pytorch extension now doing so unlocked more than 30 training speed increase for the occupancy networks and freed up basically a whole CPU to do any other thing you cannot just do training with just videos of course you need some kind of a ground Truth and that is actually an interesting problem as well the objective for storing your ground truth is that you want to make sure you get to your ground truth that you need in the minimal amount of file system operations and load in the minimal size of what you need in order to optimize for aggregate

1:09:45cross cluster throughput because you should see a compute cluster as one big device which has internally fixed constraints and thresholds so for this we rolled out a format that is native to us that's called small we use this for our ground truth our feature cache and any inference outputs so a lot of tensors that are in there and so just the cartoon here let's say these are your is your table that you want to store then that's how that would look out if you rolled out on disk so what you do is you take anything you'd want to index on so for example video timestamps you put those all in the header so that in your initial header read you know exactly where to go on

1:10:23disk then if you have any tensors you're going to try to transpose the dimensions to put a different dimension last as the contiguous Dimension and then also try different types of compression then you check out which one was most optimal and then store that one this is actually a huge step if you do feature caching unintelligible output from the machine Learning Network rotate around the dimensions a little bit you can get up to 20 increase in efficiency of storage then when you store that we also ordered the columns by size so that all your small columns and small values are together so that when you seek for a single value you're likely to overlap

1:11:03with a read on more values which you'll use later so that you don't need to do another file system operation so I could go on and on I just went on on touched on two projects that we have internally but this is actually part of a huge continuous effort to optimize the compute that we have in-house so accumulating and aggregating through all these optimizations We Now train our occupancy networks twice as fast just because it's twice as efficient and now if we add in bunch more compute and go parallel we cannot train this in hours instead of days and with that I'd like to hand it off to the biggest user of compute John

1:11:48hi everybody my name is John Emmons I lead the autopilot Vision team I'm going to cover two topics with you today the first is how we predict lanes and the second is how we predict the future behavior of other agents on the road in the early days of autopilot we modeled the lane detection problem as an image space instant segmentation task our network was super simple though in fact it was only capable of printing Lanes from of a few different kinds of geometries specifically it would segment the Eagle Lane it could segment adjacent lanes and then it had some special casing for forks and merges this simplistic modeling of the problem worked for highly structured roads like

1:12:27highways but today we're trying to build a system that's capable of much more complex Maneuvers specifically we want to make left and right turns at intersections where the road topology can be quite a bit more complex and diverse when we try to apply this simplistic modeling of the problem here it just totally breaks down

1:12:45taking a step back for a moment what we're trying to do here is to predict the spark set of Lane instances in their connectivity and what we want to do is to have a neural network that basically predicts this graph where the nodes are the lane segments and the edges encode the connectivities between these Lanes so what we have is our lane detection neural network it's made up of three components in the first component we have a set of convolutional layers attention layers and other neural network layers that encode the video streams from our eight cameras on the vehicle and produce a rich visual representation we then enhance this visual representation with a coarse Road metal

1:13:25Road level map data which we encode with a set of additional neural network layers that we call the lane guidance module this map is not an HD map but it provides a lot of useful hints about the topology of lanes inside of intersections the lane counts on various roads and a set of other attributes that help us the first two components here produced a dense tensor that sort of encodes the world but what we really want to do is to convert this dense tensor into a smart set of lanes in their connectivities we approach this problem like an image captioning task where the input is this dense tensor in the output text it's predicted into a special language that

1:14:04we developed at Tesla for encoding Lanes in their connectivities in this language of lanes the words and tokens are the lane positions in 3D space in The Ordering of the tokens in predicted modifiers in the tokens encode the connective relationships between these Lanes by modeling the task as a language problem we can capitalize on recent autoregressive architectures and techniques from the language Community for handling the multiple modality of the problem we're not just solving the computer vision problem at autopilot we're also applying the state-of-the-art and language modeling and machine learning more generally I'm not going to dive into a little bit

1:14:37more detail this language component what I have depicted on the screen here is the satellite image which sort of represents the local area around the vehicle the set of nosing edges is what we refer to as the lane graph and it's ultimately what we want to come out of this neural network we start with a blank slate we're going to want to make our first prediction here at this Green Dot the screen dots position is encoded as an index into a course grid which discretizes the 3D World now we don't predict this index directly because it would be too computationally expensive to do so there's just too many grid points and predicting a categorical distribution over this has both

1:15:15implications at training time and test time so instead what we do is we disretch the world coarsely first we predict a heat map over the possible locations and then we latch in the most probable location on this we then refine the prediction and get the precise point now we know where the position of this token is we don't know its type in this case though it's a beginning of a new Lane so we print it as a start token and because it's a star token there's no additional attributes in our language we then take the predictions from this first forward pass and we encode them using a learned additional embedding which produces a set of tensors that we

1:15:54can bind together which is actually the first word in our language of lanes we add this to the you know first position in our sentence here we then continue this process by printing the next Lane point in a similar fashion now this Lane point is not the beginning of a new Lane it's actually a continuation of the previous Lane so it's a continuation token type now it's not enough just to know that this Lane is connected to the previously protected Lane we want to encode its precise geometry which we do by regressing a set of spline coefficients we then take this Lane we encode it again and add it as the next word in the sentence we continue predicting these

1:16:34continuation Lanes until we get to the end of the prediction grid we then move on to a different Lane segment so you can see that cyan dot there now it's it's not topologically connected to that pink point it's actually forking off of that that blue sorry that green point there so it's got a fork type and Fork tokens actually point back to previous tokens from which the fork originates so you can see here the fork Point predictor is actually the index zero so it's actually referencing back to tokens that it's already predicted like you would in language we continue this process over and over again until we've enumerated all of the tokens in the lane graph and then the

1:17:13network predicts the end of sentence token yeah I just want to note that the reason we do this is not just because we want to build something complicated it's almost feels like a turing complete machine here with zero networks though is that we tried simple approaches for example trying to just segment the lanes along the road or something like that but then the problem is when there's uncertainty say you cannot see the road clearly and there could be two lanes or three lanes and you can't tell a simple segmentation based approach would just draw both of them is kind of a 2.5 Lane situation and the post processing algorithm would hilariously fail when the predictions

1:17:48are such yeah the problems don't end there I mean you need to predict these connective con like these connective Lanes inside of intersections which it's just not possible with the approach that ashok's mentioning which is why we had to upgrade to this sort of like overlaps like this segmentation would just go Haywire but even if you try very hard to you know put them on separate layers it's just a really hard problem but language just offers a really nice framework for modern getting a sample from a posterior as opposed to you know trying to do all of this in post-processing but this doesn't actually stop for just autopilot right John this can be used

1:18:20for Optimus yeah and you know I guess they wouldn't be called Lanes but you could imagine you know sort of in this you know stage here that you might have sort of paths that sort of you know encode the possible places that people could walk yeah it's basically if you're in a factory or in a you know home setting you can just ask the robot okay let me please talk to the kitchen or please route to some location in the factory and then we predict a set of Pathways that would you know go through the aisles take the robot and say okay this is how you get to the kitchen it just really gives us a nice framework to model these different paths that simplify the navigation problem or the

1:18:54downstream planner all right so ultimately what we get from this Lane detection network is a set of lanes in their connectivities which comes directly from the network there's no additional step here for specifying these you know dense predictions into into indispersed ones this is just a direct unfiltered output of the network

1:19:16okay so I talked a little bit about Lanes I'm going to briefly touch on how we model and predict the future paths in other semantics on objects so I'm just going to go really quickly through two examples the video on the right here we've got a car that's actually running a red light and turning in front of us what we do to handle situations like this is we predict a set of short time Horizon future trajectories on all objects we can use these to anticipate the dangerous situation here and apply whatever you know braking and steering action is required to avoid a collision in the video on the right there's two vehicles in front of us the one on the

1:19:50left lane is parked apparently it's being loaded unloaded I don't know why the driver decided to park there but the important thing is that our neural network predicted that it was stopped which is the red color there the vehicle in the other lane as you notice also is stationary but that one's obviously just waiting for that red light to turn green so even though both objects are stationary and have zero velocity it's the semantics that is really important here so that we don't get stuck behind that awkwardly parked car predicting all of these agent attributes presents some practical problems when trying to build a real-time system we need to maximize the frame rate of

1:20:24our object section stack so that autopilot can quickly react to the changing environment every millisecond really matters here to minimize the entrance latency our neural network is split into two phases in the first phase we identified locations in 3D space where agents exist in the second stage we then pull out tensor's ethos 3D locations append it with additional data that's on the vehicle and then we you know do the rest of the processing this specification step allows the neural number to focus compute on the areas that matter most which gives us Superior performance for a fraction of the latency cost so putting it all together the autopilot Vision stack predicts more

1:21:02than just the geometry and kinematics of the world it also predicts a rich set of semantics which enables safe and human-like driving I'm not going to hand things off to streak we'll tell us how we run all these cool neural networks on our FSD computer thank you [Applause] hi everyone I'm SRI today I'm going to give glimpse of what it takes to run this FSC networks in the car and how do we optimize for the inference latency today I'm going to focus just on FSD Lanes Network that John just talked about so when we started this track we wanted to know if we can run this FSC Lanes Network natively on the trip engine which is our in-house neural network

1:21:48accelerator that we built in the FSD computer when we build this Hardware we kept it simple and we made sure it can do one thing ridiculously fast dense dot products but this architecture is auto regressive and iterative where it crunches through multiple attention attention blocks in the Inner Loop producing sparse points directly at every step so the challenge here was how can we do this parse Point prediction and sparse computation on a dense dot product engine let's see how we did this on the trip so the network predicts the heat map of most probable spatial locations of the point now we do a Arc Max and a one heart operation which gives the one hard

1:22:35encoding of the index of the spatial location now we need to select the embedding associated with this index from an embedding table that is learned during training to do this on trip we actually built a lookup table in SRAM and we engineered the dimensions of this embedding such that we could achieve all of this thing with just matrix multiplication not just that we also wanted to store this embedding into a token cache so that we don't recompute this for every iteration rather reuse it for future Point prediction again we pulled some tricks here where we did all these operations just on the dot product engine it's actually cool that our team

1:23:19found creative ways to map all these operations on the trip engine in ways that were not even imagined when this Hardware was designed but that's not the only thing we have to do to make this work we actually implemented a whole lot of operations and features to make this model compilable to improve the intake accuracy as well as to optimize performance all of these things helped us run this 75 million parameter model just under 10 millisecond of latency consuming just 8 watts of power but this is not the only architecture running in the car there are so many other architectures modules and networks we need to run in the car to give a sense of scale there are about

1:24:05a billion parameters of all the networks combined producing around 1000 neural network signals so we need to make sure we optimize them jointly and such that we maximize the compute utilization throughput and minimize the latency so we built a compiler just for neural networks that shares the structure to traditional compilers as you can see it takes the massive graph of neural Nets with 150k nodes and 375k connection takes this thing partitions them into independent subgraphs and com compels each of those sub graphs natively for the inference devices then we have a neural network Linker which shares the structure to traditional Linker where we perform this

1:24:53link time optimization then we solve an offline optimization problem for with compute memory and memory bandwidth constraints so that it comes with an optimized schedule that gets executed in the car on the runtime we designed a hybrid scheduling system which basically does heterogeneous scheduling on one SOC and distributed scheduling across both the socs to run these networks in a model parallel fashion to get 100 drops of compute utilization we need to optimize across all the layers of software right from tuning the network architecture the compiler all the way to implementing a low latency high bandwidth RDMA link across both the srcs and in fact going even deeper to

1:25:40understanding and optimizing the cache coherent and non-coherent data Paths of the accelerator in the soc this is a lot of optimization at every level in order to make sure we get the highest frame rate and as every millisecond counts here and this is this is just the this is the visualization of the neural networks that are running in the car this is our digital brain essentially as you can see these operators are nothing but just the matrix multiplication correlation to name a few real operations running in the car to train or train this network with a billion parameters you need a lot of labeled data so aegon is going to talk about how do we achieve this with the

1:26:24auto labeling pipeline [Applause] thank you hi everyone I'm Jurgen Zhang and I'm leading a geometric Vision at autopilot so yeah let's talk about Auto labeling so we have several kinds of Auto labeling Frameworks to support various types of networks but today I'd like to focus on the awesome Lanes net here so to successfully train and generalize this network to everywhere we think we went tens of millions of trips from probably one minute one million intersection or even more so then how to do that so it is certainly achievable to Source sufficient amount of trips because we already have as Tim explained earlier we already have like 500 000 trips per day cash rate

1:27:26however converting all those data into a training form is a very challenging technical problem to solve this challenge we've tried various ways of manual and auto labeling so from the First Column to the second from the second to the third each Advance provided us nearly 100 x Improvement in throughput but still we run an even better Auto labeling machine that can provide provide providers good quality diversity and scalability we made all these requirements despite the huge amount of engineering effort required here we've developed a new auto labeling machine powered by multi-trip reconstruction so this can replace 5 million hours of manual labeling with just 12 hours on

1:28:18cluster for labeling 10 000 trips so how we solved there are three big steps the first step is high Precision trajectory and structure recovery by multi-camera visual inertia odometry so here all the features including ground surface are inferred from videos by neural networks then tracked and reconstructed in the vector space so the typical drift rate of this trajectory in car is like 1.3 centimeter per meter and 0.45 Milli radian per meter which is pretty decent considering its compact compute requirement than the recovery service and raw details are also used as a strong guidance for the later manual verification step this is also enabled in every FSD

1:29:05vehicle so we get pre-processed trajectories and structures along with the trip data the second step is multitude reconstruction which is the big and core piece of this machine so the video shows how the previously shown trip is reconstructed and aligned with other trips basically other trips from different people not the same vehicle so this is done by multiple internet steps like course alignment pairwise matching joint optimization then further surface refinement in the end the human analyst comes in and finalizes the label so each happy steps are already fully parallelized on the cluster so the entire process usually takes just a couple of hours

1:29:55the last step is actually Auto labeling the new trips so here we use the same multi-trip alignment engine but only between pre-built reconstruction and each new trip so it's much much simpler than fully reconstructing all the clips altogether that's why it only takes 30 minutes per trip to Auto label instead of manual

1:30:19several hours of manual labeling and this is also the key of scalability of this machine this machine easily scales as long as we have available compute and trip data so about 50 trees were newly order labeled from this scene and some of them are shown here so 53 from different vehicles so this is how we capture and transform the space-time slices of the world into the network supervision yeah one thing I like to note is that Legend just talked about how we Auto label our lanes but we have Auto laborers for almost every task that we do including our planner and many of these are fully automatic like no humans involved for example for objects or

1:31:06other kinematics the shapes their Futures everything just comes from Auto labeling and the same is true for occupancy too and we have really just built a machine around this yeah so if you can go back one slide but one more it says parallelized on cluster so so that sounds pretty straightforward but it really wasn't maybe it's it's fun to share how something like this comes about so a while ago we didn't have any auto labeling at all and then someone makes a script it starts to work it starts working better until we reach a volume that's pretty high and we clearly need a solution and so there were two other engineers in our team who were like you know that's

1:31:46an interesting you know thing what we needed to do was build a whole graph of essentially python functions that we need to run one after the other first you pull the clip then you do some cleaning then you do some Network inference then another Network inference until you finally get this but so you need to do this as a large scale so I so I tell them we probably need to shoot for you know 100 000 Clips per day or like 100 000 items that seems good and so the engineers said well we can do you know a bit of postgres and a bit of elbow grease we can do it meanwhile we are a bit later and we're doing 20 million of these functions every single day

1:32:26again we pull in around half a million clips and on those we run a ton of functions each of these in a streaming fashion and so that's kind of the back end infra that's also needed to not just run training but also Auto labeling yeah it really is like a factory that produces labels and like production lines yield quality inventory like all of the same Concepts applied to this label Factory that applies for you know the factory for our cars that's right

1:32:53okay thanks Ashok so yeah so concluding this section I'd like to share a few more challenging and interesting examples for Network for sure and even for humans probably so from the top there's like examples for like lack of Lights case or foggy night or roundabout and occlusions by heavy occlusions by parked cars and even rainy night with the raindrops on camera lenses these are challenging but once their original scenes are fully reconstructed by other clips they all of them can be Auto labeled so that our cards can drive even better through these challenging scenarios so now let me pass the mic to David to learn more about how Sim is creating the

1:33:36new world on top of these labels thank you thank you jagan my name is David and I'm going to talk about simulation so simulation plays a critical role in providing data that is difficult to source and or hard to label however 3D scenes are notoriously slow to produce take for example the simulated scene playing behind me a complex intersection from Market Street in San Francisco it would take two weeks for artists to complete and for us that is painfully slow however I'm going to talk about using jaegan's automated ground truth labels along with some brand new tooling that allows us to procedurally generate this scene and many like it in just five

1:34:24minutes that's an amazing a thousand times faster than before so let's dive in to our scene like this is created we start by piping the automated ground truth labels into our simulated World creator tooling inside the software Houdini starting with Road boundary labels we can generate a solid Road mesh and re-topologize it with the lane graph labels this helps inform important Road details like Crossroad slope and detailed material blending next we can use the line data and sweep geometry across its surface and project it to the road creating Lane paint decals next using median edges we can spawned Island geometry and populate it with randomized foliage this drastically

1:35:11changes the visibility of the scene now the outside world can be generated through a series of randomized heuristics a modular building generators create visual obstructions while randomly placed objects like hydrants can change the color of the curbs while trees can drop leaves below it obscuring lines or edges next we can bring in map data to inform positions of things like traffic traffic lights or stop signs we can trace along its normal to collect important information like number of lanes and even get accurate street names on the signs themselves next using Lane graph we can determine Lane connectivity and spawn directional Road markings on the road and they're

1:35:52accompanying road signs and finally with Lane graph itself we can determine Lane adjacency and other useful metrics to spawn randomized traffic permutations inside our simulator and again this is all automatic no artists in the loop and happens within minutes and now this sets us up to do some pretty cool things since everything is based on data and heuristics we can start to fuzz parameters to create visual variations of the single ground truth it can be as subtle as object placement and random material swapping to more drastic changes like entirely new biomes or locations of environment like Urban Suburban or rural this allows us to create infinite targeted permutations

1:36:36for specific ground truths that we need more ground Truth for and all this happens within a click of a button and we can even take this one step further by altering our ground truth itself say John wants his Network to pay more attention the directional Road markings to better detect an upcoming captive left turn lane we can start to procedurally alter our lane graph inside the simulator to help Focus to create entirely new flows through this intersection to help Focus the Network's attention to the road markings to create more accurate predictions and this is a great example of how this tooling allows us to create new data that could never be collected from The

1:37:16Real World and the true power of this tool is in its architecture and how we can run all tasks in parallel to infinitely scale so you saw the tile Creator tool and action converting the ground truth labels into their counterparts next we can use our tile extractor tool to divide this data into geohash tiles about 150 meters square in size we then save out that data into separate geometry and instance files this gives us a clean source of data that's easy to load and allows us to be rendering engine agnostic for the future then using a tile loader tool we can summon any number of those cache tiles using a geohash ID currently we're doing about these five by five tiles or three

1:38:03by three usually centered around Fleet hotspots or interesting land graph locations in the tile loader also converts these tile sets into U assets for consumption by the Unreal Engine and gives you a finished project product from what you saw in the first slide and this really sets us up for size and scale as you can see on the map behind us we can easily generate most of San Francisco city streets and this didn't take years or even months of work but rather two weeks by one person we can continue to manage and grow all this data using our PDG Network inside of the tooling this allows us to throw compute at it and regenerate all these tile sets overnight

1:38:47this ensures all environments are of consistent quality and Futures which is super important for training since new ontologies and signals are constantly released and now to come full circle because we generated all these tile sets from ground truth data that contain all the weird intricacies from The Real World and we can combine that with the procedural Visual and traffic variety to create Limitless targeted data for the network to learn from and that concludes the Sim section I'll pass it to Kate to talk about how we can use all this data to improve autopilot thank you

1:39:34thanks David hi everyone my name is Kate Park and I'm here to talk about the data engine which is the process by which we improve our neural networks via data we're going to show you how we deterministically solve interventions via data and walk you through the life of this particular clip in this scenario autopilot is approaching a turn and incorrectly predicts that Crossing vehicle as stopped for traffic and thus a vehicle that we would slow down for in reality there's nobody in the car it's just awkwardly parked we built this tooling to identify the mispredictions correct the label and categorize this clip into an evaluation set this particular clip happens to be one of 126

1:40:18that we've diagnosed as challenging parked cars at turns because of this infra we can curate this evaluation set without any engineering resources custom to this particular challenge case to actually solve that challenge case requires mining thousands of examples like it and it's something Tesla can trivially do we simply use our data sourcing infra request data and use the tooling shown previously to correct the labels by surgically targeting the mispredictions of the current model we're only adding the most valuable examples to our training set we surgically fix 13 900 clips and

1:41:01because those were examples where the current model struggles we don't even need to change the model architecture a simple way update with this new valuable data is enough to solve the challenge case so you see we no longer predict that Crossing vehicle as stopped as shown in Orange but parked as shown in red in Academia we often see that people keep data constant but at Tesla it's very much the opposite we see time and time and again that data is one of the best if not the most deterministic lever to solving these interventions we just showed you the data engine Loop for one challenge case namely these parked cars at turns but there are many

1:41:41challenge cases even for one signal of vehicle Movement we apply this data engine Loop to every single challenge case we've diagnosed whether it's buses curvy roads stopped Vehicles parking lots and we don't just add data once we do this again and again to perfect the semantic in fact this year we updated our vehicle movement signal five times and with every weight update trained on the new data we push our vehicle movement accuracy up and up this data engine framework applies to all our signals whether they're 3D multi-cam video whether the data is human labeled Auto labeled or simulated whether it's an offline model or an online model model

1:42:24and Tesla is able to do this at scale because of the fleet Advantage the infra that our engine team has built and the labeling resources that feed our Networks to train on all this data we need a massive amount of compute so I'll hand it off to Pete and Ganesh to talk about the dojo supercomputing platform thank you [Applause] thank you Katie

1:42:52thanks everybody thanks for hanging in there we're almost there my name is Pete Bannon I run the custom silicon and low voltage teams at Tesla and my name is Ganesh venkat I run the dojo program [Applause] thank you I'm frequently asked why is a car company building a super computer for training and this question fundamentally misunderstands the nature of Tesla at

1:43:25its heart Tesla is a hardcore technology company all across the company people are working hard in science and engineering to advance the fundamental understanding and and methods that we have available to build cars Energy Solutions robots and anything else so can we we can do to improve The Human Condition around the world it's a super exciting thing to be a part of and it's a privilege to run a very small piece of it in the semiconductor group tonight we're going to talk a little bit about dojo and give you an update on what we've been able to do over the last year but before we do that I wanted to give a little bit of background on the initial design that we

1:44:06started a few years ago when we got started the goal was to provide a substantial Improvement to the training latency for our autopilot team some of the largest neural networks they train today run for over a month which inhibits their ability to rapidly explore Alternatives and evaluate them so you know a 30X speed up would be really nice if we could provide it at a cost competitive and energy competitive way to do that we wanted to build a chip with a lot of arithmetic arithmetic units that we could utilize at a very high efficiency and we spent a lot of time studying whether we could do that using DRM various packaging ideas all of

1:44:49which failed and in the end even though it felt like an unnatural act we decided to reject dram as the primary storage medium for this system and instead focus on SRAM embedded in the chip SRAM provides unfortunately a modest amount of capacity but extremely high bandwidth and very low latency and that enables us to achieve High utilization with the arithmetic units those choices that particular choice led to a whole bunch of other choices for example if you want to have virtual memory you need page tables they take up a lot of space we didn't have space so no virtual memory we also don't have interrupts the accelerator is a Bare Bones Rob piece of

1:45:32Hardware that's presented to a compiler in the compiler is responsible for scheduling everything that happens in a terministic way so there's no need or even desire for interrupts in the system we also chose to pursue model parallelism as a training methodology which is not the typical situation most most machines today use data parallelism which consumes additional memory capacity which we obviously don't have so all of those choices led us to build a machine that is pretty radically different from what's available today we also had a whole bunch of other goals one of the most important ones was no limits so we wanted to build a compute fabric that would scale in an unbounded

1:46:17way for the most part I mean obviously there's physical limits now and then but you know pretty much if your model was too big for the computer you just had to go buy a bigger computer that's what we were looking for today the way package machines are packaged there's a pretty fixed ratio of for example gpus CPUs and and dram capacity and network capacity and we really wanted to disaggregate all that so that as models evolved we could vary the ratios of those various elements and make the system more flexible to meet the needs of the autopilot team yeah and it's so true with like No Limits philosophy was our guiding star all the way all of our choices were

1:47:01centered around that and and to the point that we didn't want traditional data center infrastructure to limit our capacity to execute these programs at speed so that's why we that's why we sorry about that that's why we integrated vertically our data center entire data center by doing a vertical integration of the data center we could extract new levels of efficiency we could optimize power delivery Cooling and as well as system management across the whole data center stack rather than doing Box by box and integrating that those boxes into Data Centers and to do this we also wanted to

1:47:51integrate early to figure out limits of scale for our software workloads so we integrated Dojo environment into our autopilot software very early and we learned a lot of lessons and today Bill Chang will go over our hardware update as well as some of the challenges that we faced along the way and Rajiv kurian will give you a

1:48:16glimpse of our compiler technology as well as go over some of our cool results great there we go thanks Pete thanks Ganesh I'll start tonight with a high level vision of our system that will that will help set the stage for the the challenges and the problems we're solving and then also how software will then leverage this for performance now our vision for Dojo is to build a single unified accelerator a very large one software would see a seamless compute plane with globally addressable very fast memory and all connected together with uniform high bandwidth and low latency now to realize this we we need to use density to achieve performance

1:49:12now we leverage technology to get this density in order to break levels of hierarchy all the way from the chip to the scale out systems now silicon technology has has used this has done this for decades chips have followed Moore's law to for density and integration to get performance scaling

1:49:34now a key step in realizing that Vision was our training tile not only can we integrate 25 dies at extremely high bandwidth but we can scale that to any number of additional tiles by just connecting them together now last year we showcased our first functional training tile and at that time we already had workloads running on it and since then the team here has been working hard and diligently to deploy this at scale now we've made amazing progress and had a lot of Milestones along the way and of course we've had a lot of unexpected challenges but this is where our fail fast philosophy has allowed us to push our boundaries now pushing density for performance

1:50:27presents all new challenges one area is power delivery here we need to deliver the power to our compute die and this directly impacts our Top Line compute performance but we need to do this at unprecedented density we need to be able to match our die pitch with a power density of almost one amp per millimeter squared and because of the extreme integration this needs to be a multi-tiered vertical power solution and because there's a complex heterogeneous material Stack Up we have to carefully manage the material transition especially CTE

1:51:10now why does the coefficient of thermal expansion matter in this case CTE is a fundamental material property and if it's not carefully managed that Stack Up would literally rip itself apart so we started this effort by working with vendors to deliver to develop this power solution but we realized that we actually had to develop this in-house now to balance schedule and risk we built quick iterations to support both our system bring up and software development and also to find the optimal design and stack up that would meet our final production goals and in the end we were able to reduce CTE over 50 percent and meet our performance by 3x over our

1:51:59initial version now needless to say finding this optimal material stack up while maximizing performance at density is extremely difficult now we did have unexpected challenges along the way here's an example where we push the boundaries of integration that led to component failures this started when we scaled up to larger and longer workloads and then intermediate intermittently a single site on a tile would fail now they started out as recoverable failures but as we pushed some much higher and higher power these would become permanent failures now to understand this failure you have to understand why and how we build our power modules solving density at every level is the is

1:52:54is the Cornerstone of actually achieving our system performance now because our X Y plane is used for high bandwidth communication everything else must be stacked vertically this means all other components other than our die must be integrated into our power modules now that includes our clock and our power supplies and also our system controllers now in this case the favors were due to losing clock output from our oscillators

1:53:25and after an extensive debug we found that the root cause was due to vibrations on the module from piezoelectric effects our nearby capacitors now singing caps are not a new phenomenon and in fact very common in power design but normally clock chips are placed in a very quiet area of the board and often not affected by power circuits but because we needed to achieve this level of integration these oscillators need to be placed in very close proximity now due to our switching frequency and then the vibration resonance created it caused Auto plane vibration on our mems oscillator that caused it to crack

1:54:10now the solution to this problem is a multi-prong approach we can reduce the vibration by using soft terminal caps we can update our mems part with a lower Q factor for the outer plane Direction and we can also update our switching frequency frequency to push the resonance further away from these sensitive bands

1:54:38now addition to the to the density at the system level we've been making a lot of progress at the infrastructure level we knew that we had to re-examine every aspect of the data center infrastructure in order to support our unprecedented power and cooling density we brought in a fully custom designed CDU to support dojo's dense cooling requirements and the amazing part is we're able to do this at a fraction of the cost versus buying off the shelf and modifying it and since our Dojo cabinet integrates enough power and cooling to match an entire row of standard it racks we need

1:55:19to carefully design our cabinet and infrastructure together and we've already gone through several iterations of this cabinet to optimize this and earlier this year we started load testing our power and cooling infrastructure and we were able to push it over two megawatts before we tripped our substation and got a call from the city yeah now last year we introduced only a couple of components of our system the custom D1 die and the training tile but we teased the exit pod as our end goal we'll walk through the remaining parts of our system that are required to build out this exit pod now the system tray is a key part of realizing our vision of a single

1:56:06accelerator it enables us to seamlessly seamlessly connect tiles together not only within the cabinet but between cabinets we can connect these Tiles at very tight spacing across the entire accelerator and this is how we achieve our uniform communication this is a laminated bus bar that allows us to integrate very high power mechanical and thermal support in an extremely dense integration it's 75 millimeters in height and and supports six Tiles at 135 kilograms this is the equivalent of three to four fully loaded high performance racks

1:56:49next we need to feed data to the training tiles this is where we've developed the dojo interface processor it provides our system with high bandwidth dram to Stage our training data and it provides full memory bandwidth to our training tiles using TTP our custom protocol that we can use to communicate across our entire accelerator it also has high speed ethernet that helps us extend this custom protocol over a standard ethernet and we provide native Hardware support for this with little to no software overhead and lastly we can connect connect to it through a standard Gen 4 pcie interface

1:57:34now we pair 20 of these cards per tray and that gives us 640 gigabytes of high bandwidth dram and this provides our disaggregated memory layer for our training tiles these cards are a high bandwidth ingest path both through pcie and ethernet they also provide a high ratex Z conductivity path that allows shortcuts across our large Dojo accelerator

1:58:02now we actually integrate the host directly underneath our system tray these hosts provide our ingest processing and connect to our interface processors through pcie these hosts can provide Hardware video decoder support for video Based training and our user applications land on these hosts that we so we can provide them with the standard x86 Linux environment

1:58:33now we can put two of these assemblies into one cabinet and pair it with redundant power supplies that do direct conversion of three-phase 480 volt AC power to 52 volt DC power Now by focusing on density at every level we can realize the vision of a single accelerator now starting with the uniform nodes on our custom D1 die we can connect them together in our fully integrated training tile and then finally seamlessly connecting them across cabinet boundaries to form our Dojo accelerator and all together we can house two full accelerators in our exopod for a combined one exaflop of ml compute now all can altogether this amount of technology and integration has only ever

1:59:29been done a couple of times in the history of compute next we'll see how software can leverage this to accelerate their performance [Applause] thanks Bill my name is Rajiv and I'm going to talk some numbers so our software stack begins with the pi torch extension that speaks to our commitment to one standard pytorch models out of the box we're going to talk more about our jit compiler and the ingest pipeline that feeds the hardware with data abstractly performances tops times utilization times accelerator occupancy we've seen how the hardware provides Peak Performance is the job of the compiler to extract utilization from the hardware while code is running on it

2:00:19and as the job of the ingest pipeline to make sure that data can be fettered at throughput high enough for the hardware to not ever starve so let's talk about why communication bound models are difficult to scale but before that let's look at why resnet 50 like models are easier to scale you start off with a single accelerator run the forward and backward passes followed by the optimizer than to scale this up you run multiple copies of this on multiple accelerators and while the gradient is produced by the backward pass do need to be reduced and this introduces some communication this can be done Pipeline with the backward pass

2:00:58this setup scales fairly well almost linearly for models with much larger activations we run into a problem as soon as we want to run the forward pass the batch size that fits in a single accelerator is often smaller than the batch Norm surface so to get around this researchers typically want this setup on multiple accelerators in sync batch Norm mode this introduces latency bound communication to the critical path of the forward pass and we already have a communication bottleneck and while there are ways to get around this they usually involve tedious manual work best suited for a compiler and ultimately there's no skirting around the fact that if your state does

2:01:37not fit in a single accelerator you can be communication about and even with significant efforts from our ml Engineers we see such models don't scale linearly the dojo system was built to make such models work at high utilization the high density integration is was built to not only accelerate the compute bound portions of a model but also the latency bound portions like a batch Norm or the bandwidth bound portions like a gradient all reduced or a parameter all gather a slice of the dojo mesh can be carved out to run any model the only thing users need to do is to make the slice large enough to fit a bathroom surface for their particular model

2:02:23after that the partition presents itself as one large accelerator freeing the users from having to worry about the internal details of execution and as the job of the compiler to maintain this abstraction fine-grained synchronization Primitives in uniform low latency makes it easy to accelerate all forms of parallelism across integration boundaries tensors are usually store sharded in SRAM and replicated just in time for layers execution we depend on the high Dojo bandwidth to hide this replication time tensor replication and other data transfers are overlapped with compute and the compiler can also recompute layers when it's profitable to do so

2:03:05we expect most models to work out of the box as an example we took the recently released stable diffusion model and got it running on dojo in minutes out of the box the Kampala was able to map it in a model parallel manner on 25 Dojo dies here's some pictures of a cyber truck on Mars generated by stable diffusion running on dojo looks [Applause]

2:03:33looks like it still has some ways to go before matching the Tesla Design Studio team so we've talked about how communication bottlenecks can hamper scalability perhaps an acid test of a compiler and the underlying Hardware is executing a cross-diabash form layer like mentioned before this can be a Serial bottleneck the communication phase of a bachelor begins with nodes Computing the local mean and standard deviations then coordinating to reduce these values then broadcasting these values back and then they resume their work in parallel so what would an ideal bathroom look like on 25 Dojo dots let's say the previous less activations are already split across dice

2:04:17we would expect that 350 nodes on each die to coordinate and produce dilocal mean and standard deviation values ideally these would get would further reduce but the final value ending somewhere and towards the middle of the tile we would then hope to see a broadcast of this value radiating from the center let's see how the compiler actually executes a real Bachelor operation across 25 dice the communication trees were extracted from the compiler and the timing is from a real Hardware one we're about to see 8750 nodes on 25 dies coordinating to reduce and then broadcast the bastrum mean and standard deviation values dial Local reduction followed by global reduction towards the

2:05:02middle of the tie then the reduced value broadcast radiating from the middle accelerated by the Hardware's broadcast facility this operation takes only five microseconds on 25 Dojo dice the same operation takes 150 microseconds on 24 gpus this is an orders of magnitude improvement over gpus and while we talked about an already separation in the context of a batch Norm it's important to reiterate that the same advantages apply to all other communication Primitives and these Primitives are essential for large-scale training so how about full model performance so while we think that resnet 50 is not a good representation of real world Tesla workloads it is a standard

2:05:49Benchmark so let's start there we are already able to match the 100 die for die however perhaps a hint of dojo's capabilities is that we're able to hit this number with just a batch of eight per die but Dojo was really built to tackle larger complex models so when we set out to tackle real world workloads we looked at the usage patterns of our current GPU cluster and two models stood up the auto labeling networks a class of offline models that are used to generate ground truth and the occupancy networks that you heard about the auto labeling networks are large models that have high arithmetic intensity while the occupancy networks can be in justbound

2:06:29we chose these models because together they account for a large chunk of our current GPU cluster usage and they would challenge the system in different ways so how do we do on these two networks the results we're about to see were measured on multi-die systems for both the GPU and Dojo but normalized to per die numbers on our Auto labeling Network we're already able to surpass the performance of an a100 with our current Hardware running on our older generation vrms on our production Hardware with our new vrms that translates to doubling the throughput of an a100 and our model showed that with some key compiler optimizations we could get to more than 3x the performance of an a100

2:07:13we see even bigger leaps on the occupancy Network almost 3x with our production Hardware with room for more foreign [Applause] level of compiler performance we could replace the ml compute of one two three four five and six GPU boxes with just a single Dojo tile [Applause] and this Dojo tile costs less than one of these GPU boxes whoa yeah what it really means is that networks that took more than a month to train now take less than a week

2:08:10alas when we measure things it did not turn out so well at the pie torch level we did not see our expected performance out of the git and this timeline chart shows our problem the teeny tiny little green bars that's the compile code running on the accelerator the row is mostly white space where the hardware is just waiting for data

2:08:33with our dense ml compute Dojo hosts effectively have 10x more ml compute than the GPU host the data loaders running on this one host simply couldn't keep up with all that ml Hardware so to solve our data loader scalability issues we knew we had to get over the limit of this single host the Tesla transport protocol moves data seamlessly across host tiles and ingest processors so we extended the Tesla transport protocol to work over ethernet we then built the dojo network interface called the dnic to leverage TTP over ethernet this allows any host with a dnic card to be able to DM it to and from other TTP endpoints so we started with the dojo mesh

2:09:17then we added a tier of data loading hosts equipped with the dnic card we connected these hosts to the mesh via an ethernet switch now every host in this data loading tier is capable of reaching all TTP endpoints in the dojo mesh via Hardware accelerated dma after these optimizations went in our occupancy went from four percent to 97 percent so the data loading sections have reduced

2:09:51the data loading sections have reduced drastically and the ml Hardware is kept busy we actually expect this number to go to 100 pretty soon after these changes went in we saw the full expected speed up from the pytorch layer and we were back in business so we started with Hardware design that breaks through traditional integration boundaries in service of our vision of a single giant accelerator we've seen how the compiler and just layers build on top of that Hardware so after proving your performance on these complex real-world networks we knew what our first large-scale deployment would Target our high arithmetic intensity Auto labeling Networks

2:10:30today that occupies 4000 gpus over 72 GPU racks with our dense compute and our high performance we expect to provide the same throughput with just four Dojo cabinets [Applause] and these four Dojo cabinets will be part of our first exopod that we plan to build by quarter one of 2023 this one more than double Tesla's Auto labeling capacity

2:11:05[Applause] the first exopart as part of a total of seven extra parts that we plan to build in Palo Alto right here across the wall [Applause] and we have a display cabinet from one of these exopods for everyone to look at six tiles densely packed on a tray 54 petaflops of compute 640 gigabytes of high bandwidth memory with power and host defeated a lot of

2:11:42and we're building out new versions of all our cluster components and constantly improving our software to hit new limits of skill we believe that we can get another 10x improvement with our next Generation Hardware and to realize their ambitious goals we need the best software and Hardware Engineers so please come talk to us or visit tesla.com AI thank you [Applause]

2:12:17all right all right let me know all right so we hopefully that was enough detail and now we can move to questions and guys like I think the team came out kind of come out on stage and but we really wanted to show the the depth and breadth of Tesla in artificial intelligence compute Hardware robotics actuators and

2:12:57and try to really shift the perception of the company away from you know a lot of people think we're like just a car company or we make cool cars whatever but they don't have most people have no idea that Tesla is arguably the the leader in real world AI hardware and software and that we're building what is arguably the first some of the most radical computer architecture since the the crayon supercomputer and I think if you're interested in developing some some of the most advanced technology in the world that's going to really affect the world in a positive way tells us the place to be so yeah let's fire away with some questions I think there's a mic at the front and a

2:13:48mic at the back on this side just throw mics at people yeah we jump all for the mic yeah I was impressed very much by Optimus but I wonder why they don't driven the hunt why did you choose a tender driven approach for the hunt because tendons are not very durable and why spring loaded yeah this is pretty cool awesome yes that's a great question you know when it comes to any type of actuation scheme there's trade-offs between you know whether or not it's a tendon urine system or some type of linkage based system I'd just keep in mind close to your mouth a little bit closer yeah hear me cool so yeah the main reason why we went for a tendon-based system is that you

2:14:46know first we actually investigated some synthetic tendons but we found that metallic boating cables are you know a lot stronger one of the advantages of these cables is that it's very good for part reduction we do want to make a lot of these hands so having a bunch of parts a bunch of small linkages ends up being you know a problem when you're making a lot of something one of the big reasons that you know tendons are better than linkages in a sense is that you can be anti-backlash so anti-backlash essentially you know allows you to not have any gaps or you know stuttery Motion in your fingers spring-loaded mainly what spring loaded allows us to do is it allows us to have

2:15:29active opening so instead of having to have two actuators to drive the fingers closed and then open we have the ability to you know have the tendon drive them closed and then the Springs passively extend and this is something that's seen in our hands as well right we have the ability to actively flex and then we also have the ability to extend yeah I mean our goal with Optimus is to have a robot that is maximally useful as quickly as possible so there's a lot of ways to solve the various problems of a humanoid robot and we're probably not balking up the right Tree on on all the Technical Solutions and I should say that we're we're open to evolving the Technical

2:16:10Solutions that you see here over time we're not they're not locked in stone but we do we have to pick something in and we want to pick something that's going to allow us to produce the robot as quickly as possible and have it like I said be useful as quickly as possible we're trying to follow the goal of fastest path to a useful robot that can be made at volume and we're going to test the robot internally at Tesla in in our Factory and and just see like how useful is it because you have to have a you're going to close the loop on reality to confirm that the robot is in fact useful and yeah so we're gonna use it to

2:16:54build things and we're confident we can do that with the hand that we have currently designed but this I'm for sure they'll be had version two version three and we may change the architecture quite significantly over time sorry Hi you're the Optimus robot is really impressive that you did a great job bipedal robots are really difficult but what I noticed might be missing from your plan is to acknowledge the

2:17:26utility of the human spirit and I'm wondering if Optimus will ever get a personality and be able to laugh at our jokes while they've well it folds our clothes yeah absolutely I think we want to have really fun versions of Optimus and so that Optimus can both do be utilitarian and do tasks but can also be kind of like a friend and a buddy and and hang out with you and I'm sure people will think of all sorts of creative uses for this robot and you know the thing once you have the core intelligence and actuators figured out then you can actually you know put all sorts of costumes I guess on on the robot I mean you can make the robot look you can scan the robot in many different

2:18:24ways and I'm sure people will find very interesting ways to to yeah versions of Optimus so thanks for the great presentation I wanted to know if there was an equivalent to interventions in Optimus it seems like labeling through moments where humans disagree with what's going on is important and in a humanoid robot that might be also a desirable source of information

2:18:59yes it was hanging yeah I I think we we'll have ways to remote operate the robot and intervene when it does something bad especially when we are training the robot and bringing it up and hopefully we you know design it in a way that we can stop the robot from if it's going to hit something we can just like hold it and then it'll stop it won't like you know crush your hand or something and those are all intervention data yeah and we can learn a lot from our simulation systems too where we can check for collisions and supervise that those are bad actions yeah I mean so Optimus so we want over time to for it to be you know an Android kind of Android

2:19:39that you'd seen in in Sci-Fi movies like Star Trek the Next Generation like data but obviously we could program the robot to be less robot-like and more friendly and and you know you can obviously learn to emulate humans and and feel very natural so as as AI in general improves we

2:19:59can add that to the robot and you know it should be obviously able to do simple instructions or even into it what it is that you want so you could give it a high level instruction and then it can break that down into a series of actions and and take those actions hi yeah it's exciting to think that with the Optimus you will think that you can achieve orders of magnitude of improvement in economic output that's really exciting and when Tesla started the mission was to accelerate the Advent of renewable energy or sustainable transport so with the Optimus do you still see that mission being this mission statement of Tesla or is it going to be updated with you know

2:20:52mission to accelerate the Advent of I don't know infinite abundance or Limitless Limitless economy yeah it I mean it is not strictly speaking Optimist is not strictly speaking directly in line with accelerating sustainable energy it you know to the degree that it is more efficient at getting things done than a person it does I guess help with if you know sustainable energy but it I think the mission effectively does does somewhat broaden with the Advent of Optimus to you know I don't know making the future awesome so you know I think you look at Optimus and I know about you but I I'm excited to see what optimists will become and you know this is like

2:21:44you know if you could I mean you can tell like any given technology if are you do you want to see what it's like in a year two years three years four years five years ten I'd say for sure you definitely want to see what what's happened with Optimus whereas you know a bunch of other Technologies are you know sort of plateaued about name names here but [Laughter] you know so I think Optimus is going to be incredible in like five years ten years like mind-blowing and I'm really interested to see that happen I hope you are too oh I think I have a quick question here Justin and I was wondering like are you planning to extend like conversational capabilities for the

2:22:38robot and my second follow-up question to that is what's like the end goal what's the end goal with Optimus yeah optimists would definitely have conversational capabilities so I I you'd be able to talk to it and have a conversation and it would feel quite natural so from an end goal standpoint I'm I'm

2:23:02I don't know I think it's Gonna Keep keep evolving and I'm not sure where where it ends up but some some place interesting for sure you know we always have to be careful about the you know don't go down the Terminator path that's a you know I I thought we might maybe we should start off with a video of like the Terminator starting off with this you know skull crushing but oh that might be kind of people might take that too seriously so yeah you know we we do want Optimus to be safe so we are designing in safeguards where you can locally stop the robot and you know with like basically a localized control ROM that you can't update over the Internet which I think that's quite

2:23:51important essential frankly so like a localized stop button remote remote control something like that that cannot be changed but I mean it's definitely going to be interesting it won't be boring so okay yeah I see you today you have very attractive product with dojo and its applications so I'm wondering what's the future for Dojo platform where you would like to provide like a infra charge infrastructure and service like AWS or you'll be like a sales achieve like the Nvidia so basically what's the future because of the I say you use a seven nanometer so the developer cost is like easily over 10 million US Dollars how do you make the penis is like a penis-wise

2:24:49yeah I mean Dojo is is a is a very big computer and actually we'll be use a lot of power and needs a lot of cooling so I think it's probably going to make more sense to have Dojo operate in like Amazon web services manner than to try to sell it to someone else so the the most that would be the most efficient way to operate Dojo is just have it be a a service that you can use that's available online and that where you can train your models way faster and for less money and as the

2:25:27world transitions to software 2.0 and that's on the bingo card someone I know it has to know how to drink five tequilas so let's see software 2.0 yeah we'll use a lot of neural net training so you know it kind of makes sense that over time as there's more more neural net stuff the people will want to use and the fastest lowest cost neural net training system so I think there's a lot of opportunity in that direction

2:26:12hi my name is Ali jahanian thank you for this event it is very inspirational my question is I'm wondering what is your vision for humanit robots that understand our emotions and art and can contribute to our creativity well I think there's this you're already seeing robots that at least are able to generate very interesting art with like like Dali and Delhi too and I think we'll we'll start seeing AI that can actually generate even movies that have a that have coherence like interesting movies and tell jokes so it's quite remarkable how fast AI is advancing at many companies besides Tesla

2:27:12we're headed for a very interesting future and yeah so you guys want to comment on that yeah I guess The Optimist robot can come up with physical art not just digital art you can you know you can ask for some dance moves in text or voice and then you can produce those in the future so it's a lot of like physical heart not just digital art oh yeah yeah computers can absolutely make a physical art yeah yeah 100 yeah like dance sure play soccer or whatever you I mean it needs to get more agile but over time for sure thanks so much for the presentation for the Tesla autopilot slides I noticed that the models that you were using were heavily motivated by language models and

2:27:59I was wondering what the history of that was and how much of an improvement it gave I thought that that was a really interesting curious choice to use language models for the lane transitioning so there are sort of two aspects for why we transitioned to language modeling so the first talk talk loud and close Okay okay got it yeah so the language models help us in two ways the first way is that it lets us predict lanes that we couldn't have otherwise as a shook mentioned earlier basically when we predicted Lanes in sort of a dense 3D fashion you can only model certain kinds of lanes but we want to get those crisscrossing connections inside of intersections it's just not

2:28:36possible to do that without making it a graph prediction if you try to do this with dense segmentation it just doesn't work also the lane prediction is a multimodal problem sometimes you just don't have sufficient visual information to know precisely how things look on the other side of the intersection so you need a method that can generalize and produce you know coherent predictions you don't want to be predicting two lanes in three lanes at the same time you want to commit to one in a generative model like these language models provides that

2:29:08hi oh hi my name is Giovanni yeah thanks for the presentation that's really nice I have a question for FSD team so for the neural networks how do you test like how do you do unit tests software unit tests on that like do you have like a a bunch or I don't know mid thousands or yes cases where so the neural network that after you train it you have to pass it before you release it to as a product right yeah what's your software unit testing strategies for this basically yeah glad you asked there's like a series of tests that we have defined starting from you know unit test for the software itself but then for the neural network models we have VIP sets defined where

2:29:58you know you can Define if you just have a large test set that's not enough what we find we need like sophisticated VIP sets for different failure modes and then we curate them and grow them over the time of the product so over the years we have like like hundreds of thousands of examples where we have been failing in the past that we have curated and so we for any new model we test against the entire history of these failures and then keep adding to this test set on top of this we have Shadow modes where we ship these models in silent to the car and we get data back on where they are failing or succeeding and there's extensive QA program it's very

2:30:38hard to ship a regression there's like nine levels of filters before it hits customers but then we have really good infra to make this all efficient and I'm one of the QA testers so I QA the car yeah like a Creator yeah okay so I'm constantly in the car just being queuing like whatever the latest alpha build is that doesn't totally crash finds a lot of bugs hi great event I have a question about foundational models for I have all seen that big models that really can when you scale up with data and model parameter right from gp3 to Palm it can actually now do reasoning do you see that it's essential skinning up foundational models with data and

2:31:29size and then at least you can get a teacher model right that potentially can solve all the problems and then you distill to a student model is that how you see foundational models relevant for at 100 I mean that's quite similar to our Auto labeling model so we we don't just have models that run in the car we train models that are entirely Offline that are extremely large that can't run in real time on the car so we just run those offline on the servers producing really good labels that can then train the online networks so that's one form of desolation of these teacher student models in terms of foundation models we are building some really really large data

2:32:11sets that you know are multiple petabytes and we are seeing that some of these tasks work really well when we have these large data sets like the kinematics like I mentioned video in all the kinematics out of all the objects and up to the fourth derivative and people thought we couldn't do detection with cameras detection depth velocity acceleration and imagine how precise these have to be for these higher order derivatives to be accurate and this all comes from these kind of large data sets and large models so we are seeing the equivalent of foundation models in our own way for geometry and kinematics and things like those you want to add anything John

2:32:50yeah I'll keep it brief basically whenever we train on a larger data set we see big okay basically whenever we train on a larger data set we see big improvements in our model performance and basically whenever we initialize our networks with you know some pre-training step from some other auxiliary task we basically see improvements the self-supervised or supervised with large data sets both help a lot

2:33:15hey so at the beginning Elon said that Tesla was potentially interested in building artificial general intelligence systems given the potentially transformative impact of technology like that it seems prudent to invest in technical AGI safety expertise specifically I know Tesla does a lot of technical narrow AI Safety Research I was curious if Tesla was intending to try to build expertise in technical artificial general intelligence safety specifically well if I mean if if it starts looking like we're gonna be making a significant contribution to artificial general intelligence then then we'll for sure invest in in safety I'm a big believer in AI safety I think there

2:33:59should be an AI sort of regulatory Authority at the government level just as there is a regulatory Authority for anything that affects Public Safety so we have a regulatory Authority for aircraft and cars and sort of food and drugs and because they affect Public Safety and AI also affects Public Safety so I think and this is not really something that government I think understands yet but I think I think there should be a referee that is ensuring or doing trying to ensure Public Safety for AGI and you think of like well like what are the elements that are necessary to to create AGI like the accessible data set is extremely important and if you've got a large

2:34:50number of of cars and humanoid robots processing you know petabytes of of video data and audio data from The Real World just like humans that that's that might be the biggest data set it probably is the biggest data set because in addition to that you can obviously incrementally scan the the internet but what the internet can't quite do is is have millions or hundreds of millions of cameras in the real world like I said with audio and and and other sensors as well so so I think we probably will have the most amount of data and probably the most amount of TR of training power therefore probably we will make a contribution to AGI

2:35:45hey I noticed the semi was back there but we haven't talked about it too much I was just wondering for the semi truck what are the changes you're thinking about from a sensing perspective I imagine there's very different requirements obviously than just a car if and if you don't think that's true why is that true no I think basically you can drive a car I mean think about what drives any vehicle it's a biological neural net with with eyes with cameras essentially so if and really what what is your your primary sensors are two cameras on a slow gimbal a very slow gimbal that's that's your head so if if you know if a biological neural net with with two cameras on a slow gimbal

2:36:36can drive a semi truck then if you've got like eight cameras with continuous 360 degree Vision operating at a higher frame rate and much higher reaction rate then I think it is obvious that you should be able to drive a semi or any any vehicle much better than human hi my name is Akshay thank you for the event assuming you know Optimus would be used for different use cases and would evolve at different piece for these use cases would it be possible to sort of develop and deploy different software and Hardware components independently and deploy them you know in the in Optimus so that the overall you know feature development is faster for Optimus

2:37:25reference to the questions okay all right we did not comprehend unfortunately our neural net did not comprehend the question yeah so but next question hi I want to switch a gear to the autopilot so when you guys plan to roll out the FSD beta to countries other than us and Canada and also my next question is what's the biggest bottleneck or the technological barrier you think in the current order part of the stack and how you envision to solve that to make the autopilot is considerable better than human in terms of a performance Matrix safety assurance and the human confidence and I think you also mentioned 4V fstv whatever you are guys going to combine the highway and

2:38:14the city as a single stack and some architectural big improvements can you maybe expand a bit on that thank you well that's a whole bunch of questions well we we I we're hopeful to be able to I think from a technical standpoint FSD beta should be it should be possible to roll out FSD beta worldwide by the end of this year but we you know for a lot of countries we need regulatory approval and so we are somewhat gated by the regulatory approval in other countries

2:38:52but I you know I but I think from technical standpoint it will be ready to go to to a worldwide beta by the end of this year and there's quite a big Improvement that we're expecting to release next month that will always be especially good at assessing the velocity of fast-moving cross traffic and a bunch of other things so anyone elaborate

2:39:20yeah I guess so there used to be a lot of differences between production autopilot and the full self-driving beta but those differences have been getting smaller and smaller over time I think just a few months ago we now use the same vision only object detection stack in both FSD and in the production autopilot on all vehicles there's still a few differences the primary one being the way that we predict Lanes right now so we upgraded the modeling of Lane so that it could handle these more complex geometries like I mentioned in the talk in production autopilot we still use a simpler lane model but we're extending our current FSD beta models to work in

2:39:56all sort of Highway scenarios as well yeah and the the version of FSD beta that I drive actually does have the integrated stack so just it uses the FSD stack both in city streets and Highway and Works quite well for me but we need to validate it in all kinds of weather like heavy rain snow dust and and just make sure it's working as better than the production stack in you know across a wide range of environments but we're pretty close to that I mean I think it's I don't know maybe

2:40:34it'll definitely be before the end of the year and and maybe November Yeah in our personal drives the FSD stack on Highway drives already way better than the production stack we have and we do expect to also include the parking lot stack as a part of the FSC stack before the end of this year so that will basically bring us to you sit in the car in the parking lot and drive till the end of the parking lot at a parking spot before the end of this year yeah and and in terms of the like the the fundamental the fundamental metric to optimize against is how many miles per between a necessary intervention so just massively improving the how many miles the car can drive on in full

2:41:17autonomy before an intervention is required that is safety critical so yeah that's that's the fundamental metric that we're measuring every week and we're making radical improvements on that hi thank you hi thank you so much for the presentation very inspiring my name is Daisy I actually have a non-technical question for you I'm curious if you are back to your 20s what are some of the things you wish you knew back then what are some advice you would give to your younger self

2:42:05well I'm trying to figure out something useful to say yeah yeah joined Tesla would be one thing yeah I think just trying to try to expose yourself to as many smart people as possible and I read a lot of books you know I do that did do that though so I think this there's some Merit to just also like not being like necessarily too intense and and like enjoying the moment a bit more I would say to 20 or 20 something me just to you know stop and smell the roses occasionally would probably be a good idea

2:43:00you know it's like when we're developing the the Falcon one rocket and on on the quadriline atoll and we had this beautiful little island that we're developing the rocket on and not once that during that entire time did I even have a drink on the beach I'm like well I should have had a drink on the beach that would have been fine

2:43:24thank you very much I think you have excited all of the robotics people with with Optimus this feels very much like 10 years ago in driving but as driving has proved to be harder than it actually looked 10 years ago what do we know now that we didn't 10 years ago that would make for example AGI on a humanoid come faster well I mean it it seems to me that HEI is advancing very quickly hardly a week goes by without some significant announcement and yeah I mean

2:44:02at this point like AI seems to be able to win at almost any rule-based game it's it's able to create extremely impressive art engage in conversations that are are very sophisticated you know write essays and these these just keep improving and this is so much more so so many more talented people working on AI and the hardware is getting better I think it's just it's a AI is on a super like a strong exponential curve of of improvement independent of what we do at Tesla and obviously we'll benefit somewhat from that exponential curve of improvement with AI accessible just also happens to be very good at actuators that Motors you know Motors gearboxes controllers Power

2:45:01Electronics batteries sensors and you know really like I say that you know the biggest difference between the robot on four wheels and the robot with arms and legs is is getting the actuators right actually it's an actuators and sensors problem and obviously the you know how you control those actuators and sensors but it's a yeah actuators and sensors and how you control the actuators it's a I don't know where you have to have like the ingredients necessary to create a compelling robot and we're doing it so

2:45:40hi Elan hi you are actually bringing the humanity to the next level literally Tesla and you are bringing the humanity to the next level so you said Optimus Prime Optimus will be used in next Tesla Factory my question is will a new Tesla Factory will be fully run by Optimus program and and when can general public order a humanoid yeah I think it'll it'll you know we're going to start optimists with very simple tests in the factory you know like maybe just like loading apart like you saw in the video loading apart for you know carrying apart from one place to another or loading a part into a one of our more conventional robot cells to you know that welds the

2:46:32body together so we'll start you know just trying to how do we make it useful at all and then and then gradually expand the number of situations where it's useful and I think that that the number of situations where Optimus is useful will it will grow exponentially like really really fast in terms of when people can order one I don't know I think it's not that far away well I think you mean when can people receive one so I don't know I'm like I'd say probably within three years and not more than five years within three to five years you could probably receive an Optimus

2:47:21I feel the best way to make the progress for agis to involve as many smart people across the world as possible and given the size and resource of Tesla compared to robot companies and given the state of humanity research at the moment would it make sense for the kind of Tesla to sort of Open Source some of the simulation hardware parts I think Tesla can still be the dominant platformer where it can be something like Android OS or like iOS stuff for the entire human or research would that be something that rather than keeping the Optimus to just Tesla researchers or the factory itself can open it and let the whole world explore human research

2:48:08I think we have to be careful about Optimus being potentially used in ways that are bad because that is one of the possible things to do so I think we would you know would provide optimists where you can provide instructions to optimists but where those instructions are you know governed by some laws of robotics that you cannot overcome

2:48:38so you know not doing harm to others and it would have I think probably quite a few safety related things with with Optimus yeah so all right well we'll just take maybe a few more questions and then and then and then thank you all for coming questions one deep and one Broad on the Deep for Optimus what's the current and what's the ideal controller bandwidth and then in the broader question there's this big advertisement for the depth and breadth of the company what is it uniquely about Tesla that enables that

2:49:19anyone want to tackle the bandwidth question yeah yeah so the technical bandwidth of the coast okay for the bandwidth question you have to understand or figure out what is the task that you wanted to do and what is the free if you took a frequency transform of that task what is it that you want your limbs to do and that's where you get your bandwidth from it's not a number that you can specifically just say you need to understand your use case and that's from that's where the bandwidth comes from okay what are the broad question I don't quite remember the breadth and depth thing I can answer the breadth and depth but yeah

2:50:02are you interesting on the back of the question I think we probably will just end up increasing the bandwidth or you know which translates to the effect of dexterity and reaction time of the of the robot like you could say sage it's not one Hertz and it's maybe you don't need to go all the way to 100 Hertz but I don't know maybe 10 25 I don't know over time I think the the bandwidth will will increase quite a bit or or translated to dexterity and latency you'd want to minimize that over time yeah minimize latency maximize dexterity

2:50:46I mean in terms of breadth and depth I I guess you know we've got we're a pretty big company at this point so we've got a lot of different areas of expertise that we necessarily have to develop in order to make autonomous or in order to make electric cars and then in order to make autonomous electric cars we've we've just I mean Tesla is like a whole series of startups basically and so far they've almost all been quite successful so we must be doing something right and I you know I consider one of my core responsibilities Iranian company is to have an environment where great Engineers can flourish and and I think in a lot of companies I don't know maybe most companies if

2:51:33if somebody's a really talented driven engineer they they're unable to actually their their talents are are suppressed at a lot of companies and it's you know and and some of the companies that the engineering Talent is suppressed in a way that is maybe not obviously bad but but where it's just so comfortable and you paid so much money and you but you your the output you actually have to produce is so low that it's like a Honey Trap you know so like there's a few Honey Trap places in Silicon Valley where they're not necessarily don't seem like bad places for engineers but you have to say like a good engineer went in and what did they get out

2:52:17and the output of of that engineering Talent it seems very low even though there seem to be enjoying themselves that's why I call it there's a few Honey Trap companies in Silicon Valley Tesla is not a Honey Trap we're demanding and it's like you're going to get a lot of done and it's going to be really cool and it's not going to be easy but if you are a super talented engineer your talents will be used I think to a greater degree than

2:52:54anywhere else you know SpaceX also that way so High Line I have two questions so both to the autopilot team so the thing is like I have been following your progress for the past few years so today you have made changes on like the lean detection like you said that like previously you are doing instant semantic segmentation now you guys are built transfer models for like building the lanes so what are another some some other common challenges which you guys are facing right now like which you are solving in future as a curious engineer so that like we as a researcher can work on those start working on those and the second question is like I'm really

2:53:33curious about the data engine like you guys have like told a case like where the car is stopped so how are you finding cases which is very much similar to that from the data which you have like so little bit more on the data engine would be great so that's it okay I'll start answer the first question using occupancy Network as an example so what you saw in the presentation did not exist a year ago so we only spent one year about time I'm relationship with more than 12 occupancy Network and you have a one Foundation model actually to represent the entire physical world around everywhere and you always the condition is actually really really challenging so

2:54:16only over a year ago we're kind of like driving a 2d where if there's a war and it says curve we kind of represent with the same static Edge which is obviously you know not not ideal right there's a big difference between a curve and a wall when you drive you make different choices right so after we realize that we have to go to 3D we have to basically resync the entire problem and think about how we address that so this will be like one example of a challenges we have we have a conquer in the past year yeah to answer the question about how we actually source examples of the tricky stopped cars there's a few ways to go about this but two examples are one we

2:54:58can trigger for disagreements within our signals so let's say that parked bit flickers between parked and driving will trigger that back and the second is we can leverage more of the Shadow mode logic so if the customer ignores the car but we think we should stop for it we'll get that data back too so these are just different like various trigger logic that allows us to get those data campaigns back

2:55:23hi thank you for the amazing presentation thanks so much so there are a lot of companies that are focusing on the AGI problem and one of the reasons why it's such a hard problem is because the problem itself is so hard to Define several companies have several different definitions they focus on different things so what is Tesla how is Tesla defining the AGI problem and what are you focusing on specifically well well we're not actually specifically focused on AGI I'm simply saying that AGI is so is seems likely to be an emergent property of of what we're doing because we're creating all these autonomous cars and autonomous humanoids that are actually with an a truly

2:56:11gigantic data stream that's coming in and and being processed it's by far the most amount of real world data and and data you can't get by just searching the internet because you have to be out there in the world and interacting with people and interacting with the of the roads and and just you know it's Earth is a big place and reality is messy and complicated so so I think it's sort of like likely to just it just seems likely to be an emergent property of if you've got you know tens or hundreds of millions of autonomous vehicles and and maybe even a comparable number of humanoids maybe more than that on the humanoid front well that's just the most amount of

2:56:51beta and if that that video is being processed it just seems likely that you know the the cars will will definitely get way better than human drivers and the the humanoid robots will become increasingly indistinguishable from humans perhaps and and so then like you said you have a

2:57:18emergent property of of AGI and arguably the you know humans collectively are sort of a super intelligence as well especially as we improve the data rate between humans when you think like that seems to be way back in the early days the internet was like the internet was like Humanity acquiring a nervous system where now all of a sudden any one element of humanity could know all of the knowledge of of humans by connecting to the internet almost all the knowledge or certainly huge part of it whereas previously we would exchange information by osmosis by you know by we'd have like in order to transfer data so you would have to write a letter someone would have to carry the letter

2:58:04by person to another person and then a whole bunch of things in between and then it was like this position yeah I mean insanely slow when you think about it and even if you were in the Library of Congress you still didn't have access to all the world's information and you certainly couldn't search it and I know obviously very few people are in the Library of Congress so I mean one of the great a sort of equality elements like the internet is has been the most the biggest equalizer in history in terms of access to information or knowledge in any student of History I think would agree with this because you know you go back a thousand years there were very few books like

2:58:53like and books would be incredibly expensive but only a few people knew how to read and only if an even smaller number of people even had a book now now look at it like you you can access any book instantly you can learn anything for basically for free it's pretty incredible so you know I was asked recently what what period of History would I prefer to be at the most and my answer was right now this is the most interesting time in history and I read a lot of history so let's oh yeah let's do our best to keep that going yeah and to go back to one of the early questions I would answer like you can the the thing that's happened over time with respect to Tesla autopilot is

2:59:46that we've just the the neural Nets have gotten have gradually absorbed more and more software and in the limit of course you could simply take the videos as seen by the car and compare those to these the steering inputs from the steering wheel and pedals which are very simple inputs and it in principle you could

3:00:10train with nothing in between because that's what humans are doing with a biological neural net you could train based on video and and and what trains the video is the the moving of the the steering wheel and the pedals with no other software in between we're not there yet but it's gradually going in that direction oh all right wait last question

3:00:38are you going I think we've got a question in the front here hello they're right there I will do two questions fine over there hi thanks for such a great presentation we'll do your question last with FSD being used by so many people do you think what's the com how do you evaluate the company's risk tolerance in terms of performance statistics and do you think there needs to be more transparency or regulation from third parties as to how what's good enough and defining like thresholds for performance across some many miles sure well the you know I

3:01:19the the number one design requirement at Tesla is safety so like and that goes across the board so in terms of the mechanical safety of the car we have the lowest probability of injury of any cars ever tested by the government for just a passive mechanical safety essentially crash structure and airbags and whatnot

3:01:42we have the best the highest rating for active safety as well and I'm just going to get to the point where you the active safety is so ridiculously good it's it's it's it's like just absurdly better than a human and then with respect to autopilot we do publish this broadly speaking the statistics on miles driven with cars that have no autonomy or Tesla cars with no autonomy with kind of a hardware one Hardware two Hardware three and then the ones that are in FSD beta and we see steady improvements all along the way and you know sometimes there's this dichotomy of you know should you wait

3:02:30until the car is like I don't know three times safer than a person before deploying any technology but I think that's that's actually morally wrong at the point of which you believe that that adding autonomy reduces injury and death I think you have a moral obligation to deploy it even though you're going to get sued and blamed by a lot of people because the people whose lives you've saved don't know that their lives are saved and the people the people who who do occasionally die or get injured they definitely know or their estate does that it was you know whatever there's a problem with with autopilot that's why you have to look at the at the numbers in sort of total miles

3:03:16driven how many accidents occurred how many accidents were serious how many fatalities and you know we've got well over three million cars on the road so this it's that's a lot of miles driven every day it's not going to be perfect but what matters it is that is that it is very clearly safer than not deploying it yeah so I think last question

3:03:43I think yeah so thanks well the last question here okay yeah I got it okay hi so I do not work on Hardware so maybe the hardware team and you guys can enlighten me why is it required that there be symmetry in the design of Optimus because humans we have handedness right we are we use some set of muscles more than others over time there is wear and tear right so maybe you'll start to see some joint failures or some actuator failures more over time I understand that this is extremely pre-stage also we as humans have based so much

3:04:33fantasy and fiction over super human capabilities like all of us don't want to walk right over there we want to extend our arms and like we have all these you know a lot of fantasy Fantastical designs so considering everything else that is going on in terms of batteries and intensity of compute maybe you can leverage all those aspects into coming up with something well I don't know more interesting in terms of your the robot that you're building and I'm hoping you're able to explore those directions yeah I mean I think it would be cool to have like you know make Inspector Gadget real that would be pretty sweet so yeah I mean right now we we just

3:05:20want to make a basic humanoid what work well and our goal is fastest path to a useful humanoid robot I think this is this will ground Us in reality literally and ensure that we are doing something useful like one of the hardest things to do is to be useful to to actually and then and then to have high utility under the curve like how many people did you help time you know and how much help did you you provide to each person on average and then how many people did you help the total utility like trying to actually ship useful product that people like to a large number of people is so insanely hard it boggles the mind you know that's why I could say like

3:06:09man there's a hell of a difference between a company that has ship product and one has not sure product it's a game this is night and day and then even once you ship product can you make the cost the value of the output worth more than the cost of the input which is again insanely difficult especially with Hardware so but I think over time I think we cool to do creative things and have like eight arms and whatever and have different versions and maybe you know there'll be some Hardware like companies that are able to add things to an optimist like maybe we've you know added at a power port or something like that or attach them you can add you know add attachments to your

3:06:52Optimus like you can add them to your phone there could be a lot of cool things that could be done over time and there could be maybe an ecosystem of small companies that or big companies that make add-ons for Optimus so with that I'd just thank the team for their hard work you guys are awesome and yeah

3:07:16and thanks Anna thank you all for coming and for everyone online thanks for tuning in and I think this will be one of those great videos where you can like if you you can fast forward to the bits that you find most interesting but we try to give you a tremendous amount of detail literally so you can look at the video at your leisure and you can focus on the parts that you find interesting and skip the other parts so thank you all it's and we'll do this try to do this every year and we might do a monthly podcast even so but I think it'll be you know great to sort of bring you along for the ride and and like show you what what cool things are are happening and yeah thank you

3:08:02all right thanks [Applause]

资源与出处

相关视频

同频道、同主题或相邻年份