马斯克给Dojo团队定的成功标准,跟跑分无关。
他说,如果软件团队主动想把GPU集群关掉,Dojo才算成功。如果他们还舍不得关,那就是失败。
这句话出自2023年特斯拉AI Day的Q&A环节。台上坐着马斯克和一整排AI团队工程师,台下是研究者、开发者和媒体。问题从编译器聊到压铸机,从人形机器人的手指数量聊到超人类智能的风险。整场问答没有PPT,没有演示动画,只有一群人被追问细节。
把这些回答拼起来,能看到特斯拉在AI上的一套完整判断:什么该做,什么不该做,什么还没解决,什么已经放弃了幻想。
给Dojo定KPI的人,是软件团队
第一个问题就戳到痛处。一位做深度学习加速器研究的提问者问,Dojo要扩展到那么多节点,编译器问题解决了吗?还是说这套方案只能用在Dojo上?
Dojo工程师Bill的回答很直接:还没解决,但有信心解决。他们已经在原型硬件上跑通了网络,也有性能模型展示扩展曲线。难点在于局部性,也就是怎么让大部分计算留在本地。办法是做足够的模型并行和数据并行,把参数塞进SRAM工作集里,让数据在流水线里流动。随着处理器节点增加,本地内存和带宽也会增加,能做的事情更多。
提问者不依不饶:训练是软扩展应用,就算有高带宽互连,计算发生在不同位置的有限内存上,性能也未必能兑现。
马斯克接过话头,给了一个时间表:Dojo明年投入运营,最初的用途是训练海量视频。
他关心的指标是迭代速度,峰值算力排不上号。
创新本质上就是两件事:迭代多少次,以及每次迭代之间进步多少。如果你能缩短迭代间隔,改进速率就会好很多。一个模型训练要花两天还是两个小时,差别巨大。(马斯克)
然后就到了那句定KPI的话。Dojo成不成,验收标准是软件团队愿不愿意关掉GPU集群,跑分说了不算。
开源的问题也被问到。马斯克的回答没有情怀,只有账本:造这套系统极其昂贵,总得有人买单。如果完全开源,他想不到钱从哪来,除非大家愿意免费干活。
不过他补了一句:如果其他车企想授权使用这套系统,装到自己的车上,那会很酷。它从一开始就没打算只服务特斯拉。
仿真要接管整辆车
有人问,演示里的仿真画面太逼真了,这套东西会不会扩展到公司其他部门?
Autopilot仿真团队负责人Ian Glow回答,走向完全自动驾驶的过程中,需要仿真的东西会越来越多。现在仿真的是车辆动力学,接下来要接入电池管理系统、MCU,直到车上每一个部件。这样一来,自动驾驶仿真器对自动驾驶之外的场景也会有用。团队的计划是先把它做成通用仿真平台,但在此之前,大量精力会花在支持Optimus上。再往后,他们有一些初步想法,把仿真基础设施和这些工具交到公司外部的人手里。
台下有人不知道Optimus是什么,马斯克顺口解释:Optimus就是Tesla Bot的代号。现场笑了一声。
纯视觉,外加一点耳朵
一位研究者提到,车本身已经是一台机器人,为什么还要造人形机器人。他更关心多模态:语言和符号处理能不能帮助视觉理解,比如演示里那只狗和乘客一起跑动的场景。
马斯克的回答把边界划得很清楚:
驾驶基本上几乎完全靠视觉神经网络。人类用的是生物视觉神经网络,我们做的是硅基摄像头神经网络。(马斯克)
音频有一些,但不多。需要听见救护车之类的紧急车辆,需要和车里的人对话,如果有人冲着车喊话,车得知道那是什么意思。这些都属于实现完全自主的必要条件。
另一位提问者追问多视图几何的精度,合成出来的几何和真实几何差多少。答案是以厘米计:标准差通常在3到4厘米。
马斯克补充了目标:车的位置精度要瞄准厘米级。距离近的物体精度高得多,远的物体精度低,但远的物体也没那么重要,因为车不需要在很远的地方做决策,等它靠近了,精度自然会上来。
首要指令:别撞上它
关于跨国部署,有人问数据大部分来自美国,到了路况不同的国家怎么迁移。
团队回答,训练数据其实覆盖了大约50个国家。但推进完全自动驾驶时必须先挑一个国家,因为特斯拉在美国,就选了美国。连加拿大都不选,理由是加拿大的路有细微差别,解决难题的时候不该额外增加复杂度。先在美国解决,再外推到全世界。全球的视频数据一直在用。
计算机视觉组件本身跟国家关系不大。每个国家都有路、有路缘、有车,建的东西是通用的。
然后马斯克给出了那条贯穿全场的原则:
首要指令是不撞车,这条对每个国家都成立。哪怕高速公路上迫降了一架UFO,也别撞上去。你不需要认出它是什么,才能不撞上它。(马斯克)
这句话值得多想一会儿。它意味着感知系统的第一优先级是几何上的避障,语义识别可以晚一步。不知道那是什么,照样能绕开。
一体化压铸吃掉三分之一个车身
有人问起“外星无畏舰”那个概念,问人形机器人是不是说明特斯拉在制造上退回到了“人被低估了”的判断。
马斯克的回答是在纠正自己被过度解读的说法。特斯拉的生产体系里,有些环节几乎全自动,有些环节几乎全手动,走一遍整条产线能看到很宽的谱系。但绝大部分已经自动化了。
他真正想强调的是设计架构带来的变化:
用大型铝材高压压铸,整个后三分之一车身可以铸成一个零件。现在前三分之一也要这么干。车身线的规模会缩小60%到70%。(马斯克)
机器人跟制造需求没有直接关系。造人形机器人是因为那些零件本来就要造,既然要做,不如自己做,而且要做安全。如果特斯拉不做,别人也会做。
他还补了一句关于量产的判断:大规模制造极其困难,而且被严重低估,特斯拉已经把这套本事练得很熟。这句话跟人形机器人的成本直接相关,能不能把机器人做便宜,靠的就是这套本事。
不到一伏特,I²R最要命
散热和供电的问题被单独提出来。集群机架的排热和供电都有物理上限,怎么扩展,怎么跨数据中心部署?
Bill解释,设计非常模块化。计算瓦片上下面都做液冷,扣上去、接上电、接上冷却,就完全供电、完全散热,整套东西不到1立方英尺。
马斯克补了背后的能力迁移:特斯拉的功率电子和冷却积累很深,这些经验来自车辆动力总成,来自为功率电子开发的先进冷却方案,现在被搬到超级计算机上。排热极其重要,整个系统就是被热量卡住的。
他顺手说了个反差:
计算单元运行在不到1伏特,电压非常低,电流非常大,所以发热量很大。真正咬你的是I²R。(马斯克)
Dojo是台通用神经网络训练机
有人担心,训练硬件快了,模型会更大更复杂,车上推理的成本会跟着涨。Dojo会不会也用在车端?
团队回答,Andre展示的模型不全是给车用的,还有自动标注模型,还有其他超出车辆应用的模型,它们都会汇入车辆技术栈。Dojo会服务所有这些模型,不只是车上的推理。
马斯克把定位说清楚:Dojo第一个应用是消化视频数据做训练,训练结果再跑在车上的推理引擎里。这本身就是一次测试,看它是不是真的比GPU集群好。再往后,它是一台通用的神经网络训练计算机,但为神经网络做了深度优化。
他的论据是专用优于通用:
CPU和GPU的设计初衷从来都跟训练神经网络无关。我们让GPU在这件事上非常高效,但那从来不是它们的本意,本质上还在用模拟的方式跑。Dojo的做法是把整套东西ASIC化,为神经网络训练这一个目的而生。任何为特定目的设计的系统,都会比通用系统更好。(马斯克)
在工厂里,马斯克躲着机器学习走
有人问,别的公司用强化学习和机器学习优化数据中心功耗和内部流程,特斯拉在制造设计和工程流程里用不用机器学习?
马斯克的回答毫不含糊:
我不鼓励用机器学习,因为它真的很难。除非你不得不用,否则别用。有人说“我们想用机器学习解决这个任务”,这通常是个危险信号。99.9%的情况下你根本不需要它。(马斯克)
他的态度是,机器学习属于最后手段,做起来一点都不轻松,是极其困难的事。工厂里要尽量少用。
他留了一个例外:如果真有一个能听懂日常指令的人形机器人,情况可能会变。
车越安全,训练数据越少
仿真技术栈的细节被追问。团队介绍,最底层是传统游戏技术,光栅化、实时渲染,跟《GTA》里看到的东西很像。上面叠加实时光线追踪,这部分是前一晚刚做出来的。再往上是神经渲染,正在试各种方案。
目标是让神经渲染成为最后那颗樱桃,把仿真推到模型永远无法过拟合的程度。现在做的类似“照片真实感增强”那篇论文的思路,但特斯拉能做得更多,因为有远远更多的标注数据、更多算力,对环境的控制也更强,还有一批人能把它优化到实时运行。
马斯克解释了仿真的真实用途。真实道路数据是绝对主力,现实比任何仿真都更离奇,角落案例更多。仿真用来填补缝隙:
如果我们发现几个非常奇怪的例子,几张特别难处理的画面,为了有效训练,我们会生成上千个那个怪东西的变体,把重要的缺口补上。这一切都是为了让碰撞或受伤的概率随时间下降。(马斯克)
然后他讲了一个自我矛盾的循环。仿真在碰撞规避这类罕见场景上最有用。可车越擅长避免事故,事故就越少,训练集就越小,于是只能在仿真里让车撞上去。
他描述了具体的问法:前方5米,车速20米每秒,什么动作能让受伤概率最小。这类问题可以在仿真里大规模跑。逆行车辆这种场景偶尔发生,但不够频繁,靠真实数据攒不够样本。
关于域适应,团队说目前的做法是仿真直接出完整视频片段,包含运动学信息,马上拿去训练,但只占数据集里很小的一块,而且只用真实世界视频做评估。他们非常注意不过拟合。已经做过一件事:用车上同一套网络,重新训练它去分辨仿真和真实,帮模型做决策,这确实挡住了一些问题。随着数据规模上去,过拟合的风险会来,那时候靠神经渲染把这道墙再往外推。
Optimus先干没人愿意干的活
人形机器人的落地场景被反复问。马斯克的答案始终如一:
从无聊、重复、危险的工作开始。哪些活是人最不愿意干的,就从那里开始。(马斯克)
有人问人形这个形态适合什么应用,情感陪伴会不会进产品路线图。马斯克先开了个玩笑,希望这事别演变成反乌托邦科幻电影。然后回到字面:现阶段就是尽量字面地理解,这机器人能不能干那些没人想干的活。等这件事做成了,再谈别的。
他也没把话说死:它也可以是你的伙伴,你买一台,让它当朋友,随便你。他相信人们会想出非常有创意的用法。
手指数量的问题被单独拎出来。五根手指、还要控制重量,难度不小,可能得用线驱,线驱又带来一堆麻烦。马斯克的回答很工程化:
这只是第一版,我们先看着办。它得能做人做的事,是通用人形机器人。你当然可以只给它两根手指加一个拇指,现在我们先给五根,看看行不行。(马斯克)
握力不需要惊人,但得能用工具,能提袋子。
自动标注救了自动驾驶
Scaling law的问题被提出来。语言模型靠 scaling 出现了能力跃升,特斯拉的神经网络有没有类似现象?
团队确认:更大的网络通常表现更好,前提是有足够的数据去训练它。车端有延迟约束,部署超大网络需要创造力。但特斯拉不只训练上车模型,还有自动标注流水线,可以用任意规模的模型。他们训练了一批根本无法部署的模型,明显更大、效果更好,因为自动标注需要高得多的精度。
马斯克把自动标注的地位抬得很高:
没有自动标注,我认为我们解决不了自动驾驶问题。这是一种很奇特的蒸馏:用超大的模型加上问题的结构去做重建,再蒸馏进部署到车上的网络。我们有很多神经网络和很多任务,从一开始就不打算装进车里。(马斯克)
团队补充了一个时间维度的考虑。随着时间推移会有新的帧进来,计算应该分布到所有信息上,不能抓住单帧啃200毫秒,因为更新的帧已经在路上了。
神经网络可能把地球记进了权重
预测能力是这场问答里最玄的一段。
马斯克说,车的预测能力好得有点诡异,已经超过人类。转过弯之后的路看不到,车能高精度地预测出来。行人或骑车人被遮挡,只露出自行车一角,或者透过公交车车窗看到一点点,车能推断出他们在哪。
我们经常遇到这种情况:某个东西根本不可见,神经网络却在补全它,而且补得非常合理,有时候准得让人发毛。你会怀疑这是不是训练集里见过。极限情况下,你可以想象网络参数多到足以把整个地球记下来,那它就能给出正确答案,相当于一张高清地图烘焙进了权重里。(马斯克)
这段话可以当作一个思想实验来读。参数量足够大、数据足够多的时候,记忆和理解的边界会模糊。
HW3够用,HW4大约一年后
硬件问题被连着问了几个。有人问HW3在更大网络下是不是快到极限,扩展计算模式还有多少余量,什么时候非上HW4不可。
马斯克的判断是:HW3,也就是第一代FSD计算机,足以实现完全自动驾驶,安全水平比人类高出不少,大概200%到300%。
HW4会来,大概随Cybertruck推出,时间在一年左右,能力大约是四倍。但他把问题重新定义了一遍:
真正的问题是从比人安全300%提到1000%。路上本来就有驾驶水平参差不齐的人,我们仍然允许人开车。你不需要是全世界最好的司机才能上路。(马斯克)
摄像头的质疑也在这一串里。有人问,摄像头是对人眼的糟糕近似,人眼做的事远不止拍一串帧,有没有考虑事件相机,或者自研更灵活的摄像头。
马斯克回答,HW4会有新一代摄像头。但必须说清楚:当前摄像头的能力还没有被榨干。他有信心用现有的摄像头和计算硬件实现比人类安全得多的完全自动驾驶。剩下的差别是300%还是1000%,各层都会持续演进。
他给了一个远景:将来人们回头看,会惊讶于当年居然要自己开车。自动驾驶会变得像自动电梯一样平常。
印着停止标志的T恤
对抗攻击的问题被提出来。车上没有深度传感器,如果有人打印照片或者画个假隧道来骗RGB网络怎么办?
马斯克先讲了个《猫和老鼠》式的画面:在墙上画个隧道,车一头撞上去。然后他说,目前没怎么见到这种情况。但有一件事他亲自验证过:
如果你穿一件印着停止标志的T恤,对着车晃,车会停下来。这个我试过。(马斯克)
解决办法也直接:看到这类对抗攻击,就训练车去识别。哦,那是个人穿着印了停止标志的T恤,那大概不是真的停止标志。
不确定性来了就减速
规划里怎么处理不确定性,是另一个技术追问。是不是假设路上其他车都守规矩?
团队回答,系统会考虑多模态的未来,不会只挑一种可能。用物理和运动学去判断对方不会做出干扰本车的动作,然后才行动。有任何不确定就保守,让对方先行。但保守有上限,太保守就不实用了,所以某些时刻必须果断,果断的同时要确保对方能让、能理性应对。
另一个机制是影子模式。任何东西推给车队之前,先跑影子模式,看这个神经网络在那个具体场景下会怎么做。人类司机实际上在训练它。如果网络会向右打方向,而人向左打了,就出现了差异,要去问为什么会有这个差异。前提是没出事故,出了事故就不算数。
还有一个信号是闪烁。系统观测到闪烁,说明看不清,就减速,获取更多信息再行动。
理想情况下,车知道得越少,开得越慢。速度应该和置信度成正比。(马斯克)
他自己补了一句,现实中不完全成立,但方向是这个方向。
瓶颈是人
有人问,如果只能改进AI栈里的一个组件,仿真、数据采集、规划、控制,哪个对完全自动驾驶的提升最大?
马斯克的回答绕开了单选题:这是曲线下的面积,改进任何一个点都在抬高面积。短期内,所有网络都该用环绕视频,现在还有一些遗留网络没用上,正在快速修。
然后他把话题转向了招聘:
成功的拼图大部分都在,我们只是需要更多强的人来帮我们把事情做成。那才是真正的瓶颈。这也是我们办这场活动的原因之一,有大量工作要做。(马斯克)
Andre在旁边补了一句“说得好”。
超人类智能出现时,没人知道会发生什么
灾难性遗忘和在线学习被问到。团队说,文献里提到的问题他们没有遇到,因为基本不做持续学习。系统训练一次,微调几次,出一个稳定版本,经过大量评估,再推给车队。不做太多在线学习,也就没有遗忘问题。
但个性化会有。你可以设置驾驶风格:平时偏保守,还是想开快一点。就像对网约车司机说的话,我赶飞机,麻烦快一点;或者不着急,慢慢来。
最后一个问题关于AI对齐。模型越来越强,部署到现实世界,会不会跟人类意愿偏离,是现在就担心,还是等出了问题再说?
马斯克的回答没有回避:
我认为现在就该担心AI。我们在这里做的是很窄的AI,让车开得比人好,让人形机器人能做基本的事。一旦开始接近超级智能,那就没人知道会发生什么了,所有赌注都作废。特斯拉想做的,是让人们喜欢、而且明确有益的有用AI。(马斯克)
“所有赌注都作废”这句话,他说得很轻。
电梯曾经也需要操作员
收尾时,马斯克讲了一个电梯的故事。
电梯以前有操作员,站在里面推一个大继电器开关。操作员偶尔会累,会犯错,出过人命。后来电梯自动化了,你走进一栋100层的大楼,按个按钮,一点都不担心。
汽车会走同一条路。所有车都会自动,而且都是电动的。路上还会剩下一些汽油车和手动挡车,就像今天还有马一样。
整场问答里,马斯克说了好几次“我们只能等着看”,关于机器人的经济性,关于五根手指够不够用。这种不确定性跟他在Dojo、在HW3上表现出的笃定,是同一套判断的两面:能算清楚的地方给死数字,算不清楚的地方不装懂。
这场Q&A最实用的信息,是几条能立刻拿走的方法论,宏大叙事反而次要。
- 给新硬件定一个“用户愿意关掉旧方案”的成功标准。Dojo的验收不看跑分,看软件团队会不会主动想关GPU。任何内部工具、平台、基础设施项目都可以照这个思路设KPI,观察真实用户会不会自愿迁移,比任何benchmark都硬。
- 先在一个受限场景里解到极致,再外推。特斯拉明明有50个国家的数据,还是先把美国解透,连加拿大都往后放。做产品时,砍掉额外复杂度比增加覆盖面更能加速。
- 把“不需要识别也能正确行动”作为系统设计目标。不知道前方那是什么,也要能避开。这条原则可以迁移到风控、监控、异常检测,先保证安全边界,语义理解放在后面。
- 用超大模型做标注,用小模型做部署。自动标注流水线用无法上车的巨型网络换精度,再蒸馏进车端。任何受延迟或成本约束的线上系统,都可以考虑这条蒸馏路径。
- 对机器学习保持怀疑,除非真的没有别的办法。马斯克说99.9%的情况下不需要它。评估一个方案时,先问传统方法能不能做到,能就别上模型,省下的是长期维护成本。
- 在仿真里制造稀缺样本。真实世界越安全,罕见事故的数据越少。主动构造变体、生成上千个罕见场景,是补齐长尾的可行手段,前提是只用真实数据做评估,盯紧过拟合。
- 让系统的不确定性和它的行为强度挂钩。知道得越少,动得越慢。这条规则写进任何自动化决策系统都不亏。
- 上线前先跑影子模式,用人的行为当参照。新模型先观察不执行,跟人类操作比对差异,问清每一次分歧的原因,再谈放量。
最后一条也许是最容易被忽略的。当被问到最大的瓶颈是什么,马斯克没有说算力,没有说数据,说的是人。这场发布会本身就是一次招聘广告。
0:00and yeah great so we have to answer any questions you have about anything on the software hardware side where things are going and yeah fire away we have because the lights are like interrogation lights so we actually cannot see ah there we go great all right cool
0:37i can just okay there we go first off i mean thanks to all the presenters that was just super cool to see everything i'm just curious at a high level and this is kind of a question for really anyone who wants to take it to what extent are you interested in publishing or open sourcing anything that you do for the future
1:07well i mean it is fundamentally extremely expensive to create the system so somehow that has to be paid for i'm not sure how to pay for it if it's fully open sourced yeah unless people want to work for free but but i should say that this is if other car companies want to license it and use it in their cars that would be cool this is not intended to be just limited to tesla cars
1:45is for the dojo super computer so did you solve the compiler problem of scaling to these many nodes or is or if it is solved is it only applicable to dojo because i'm doing research in deep learning accelerators and getting the correct scalability or the distribution even in one ship is extremely difficult from the research project's perspective so i was just curious excuse me mike for bill
2:20have we solved the problem not yet are we confident we will solve the problem yes we have demonstrated networks on prototype hardware now we have models performance models showing the scaling the difficulty is as you said how do we keep the localities if we can do enough model parallel enough data parallel to keep most of the things local we just keep scaling we have to fit the parameters in our working set in our sram that we have and we flow through the pipe
2:50there's plenty of opportunities sorry as we get further scale for further processor nodes have more local memory memory trade also bandwidth we can do more things but as we see it now the applications that tesla has we see a clear path and our our modularity story means we can have different ratios different aspects created out of it i mean this is something that we chose for our applications internally
3:23sure locality portion of it given that training is such a soft scaling application even though you have all this compute and have a high bandwidth high bandwidth interconnect it it could not give you that performance because you are doing computations on limited memory at different locations so i was that's very curious to me when you said it's solved because i i just jumped onto the opportunity and would love to know more up given that how much you can open source yep yeah i guess the proof's in the pudding so we should have dojo operational next year and i think we'll we'll obviously use it for training video training it's i mean fundamentally this is about like
4:13the the primary application initially is we've got vast amounts of video and how we train vast amounts of video as efficiently as possible and also shorten the amount of time like if you're trying to train train to a task like just in general innovation is how many iterations and what is the average progress between each iteration and so if if you can reduce the time between iterations the rate of improvement is is much better so you know if it takes like sometimes a couple days for a model to train versus a couple hours that's that's a big deal but the the asset test here and you know what i've told the dojo team is like it's it's successful if the
5:03software team wants to turn off the gpu cluster but if they want to keep the gpp cluster on it's not successful so hi right over here love the presentation thank you for getting us out here loved everything especially the simulation part of the presentation i was wondering it looked very very very realistic are there any plans to maybe expand simulation to other parts of the company in any way mike hi i'm ian glow i manage the autopilot simulation team so as we go down the path to full self driving we're gonna have to simulate more and more of the vehicle currently we're simulating vehicle dynamics but we're going to need bms we're going to need the mcu we're going
5:53to need every single part of the vehicle integrated and that actually makes the autopilot simulator really useful for places outside of autopilot so i want to expand or we want to expand eventually to be in a universal simulation platform but i think before that we're going to be spinning up a lot of optimus support and then a little bit further down the line that we have some rough ideas on potentially how to get the simulation infrastructure and some of the cool things we've built into the hands of people outside of the company optimus is the code name for the teslabot oops
6:29optimus uprine [Laughter] yeah hi this is ali jahanian thank you for the great presentation and putting all of these cool things together yeah for a while i have been thinking that the car is already a robot so why not a humanity robot and i'm so happy that today you mentioned that you are going to build such thing especially i think that this can give opportunity for rays of putting multi modality together for instance we know that in the example that you are showed that there was a dog and with some passengers or running together
7:17the language and symbolic processing can really help for visualizing that so i was wondering if i could hear a little more about this type of putting modalities together including language and vision because i have been working with for instance mini gpt's and andre put out there and yeah i i didn't hear much about other modalities that's going into the car or at least in the simulation is is there any comment that you could tell us well driving is fundamentally basically almost entirely vision neural nets like we're basically it's running on a biological vision neural net and what we're doing here is a silicon camera neural net so and there are there's there is some
8:15amount of audio you know you want to hear if there's like emergency vehicles or you know i guess converse with the people in the car you know if somebody's yelling something at the at the car that car needs to understand what that is so you know all the things that are necessary for it to be fully autonomous yeah thank you
8:51hi thank you for all the great work that you've shown my question is for the team because the data that was shown was seems to be predominantly from the united states that the the fsd computer is being trained on but as it is being as it gets rolled out to different countries which have their own road systems and challenges that come with it how do you think that it's going to scale like like i'm assuming like the ground up is not a very viable solution so how does it transfer to different countries well there's we actually do train on using data from you know probably like 50 different countries but we have to pick
9:33you know in as we're trying to advance full self driving we need to pick one country and since we're located here we pick the us and then we get a lot of questions like why not even canada like well because the roads are a little different in canada different enough and so when trying to solve a hard problem you want to say like okay what's the let's not add additional complexity right now let's just solve it for the u.s and then we'll extrapolate to the rest of the world but we do use video from all around the world yeah i think a lot of a lot of what we are building is very country agnostic fundamentally all the computer vision components and so on
10:08don't care too much about country specific sort of features every you know different countries have roads and they have curbs and they have cars and everything we're building is fairly general for that yeah and there's the the prime directive is don't crash right and that's true for every country yes this is the prime directive and even right now the car is pretty good at not crashing and so just basically whatever it is don't hit it even if it's a ufo that crash landed on the highway and still don't hit it you should not need to recognize it in order to not hit it that's very important
11:00and i want to ask that when you do the photometric process multiview geometry how much of an error do you see is it like one millimeter one centimeter so i'm just if it's not confidential sorry what is the what's it what's the difference between the synthetic sure what is the difference between the synthetically created geometry to the actual geometry yeah it's usually within a couple centimeters three or four centimeters that's the standard deviation merge with different kind of modalities to bring down that error we primarily try to find scalable ways to label in some occasions we use other sentences to help benchmark but primary tries to use
11:49cameras for this system okay thanks yeah i mean i think we want to aim for the car to be positioned accurately to the sort of centimeter level you know some something on that order obviously it will depend on distance like close by things can be much more accurate than farther away things because and they would matter less because the car doesn't have to make decisions much farther away and as it comes close it will become more and more accurate exactly a lot of questions thanks everybody my question has to do with sort of ai and manufacturing it's been a while since we've heard about the alien dreadnought concept is the humanoid that's behind you guys
12:30is that kind of brought out of the production health timeline and saying that humans are underrated in that process well sometimes like some you know something that i say is taken to too much of an extreme there there are parts of the tesla system that are almost completely automated and then there are some parts that are almost completely manual and if you were to walk through the whole production system you would see a very wide range from yeah like i said fully automatic to almost completely manual but the vast majority it's most of it is is already automated
13:12so and then with the some of the design architecture changes like going to large aluminum high pressure die cast components we can take the entire rear third of the car and cast it as a single piece and now we're gonna do that the front third of the car as a single piece so the the body line drops by like 60 to 70 percent in size
13:41but yeah the the the robot is not is not prompted by specifically by manufacturing needs it's it's just that we're just obviously making the pieces that are needed for a useful humanoid robot so i guess we probably should make it and if we don't someone else would well and so i guess we should make it and make sure it's safe
14:12i should say like also manufacturing volume manufacturing is extremely difficult and underrated and we've gotten pretty good at that it's also important for that humanoid robot like how do you make the human robot not be super expensive and hi thank you for the present presentation and my question will be about skilling of dojo and in particular how do you scale the compute nodes in terms of thermal thermals and power delivery because there is only so much heat that you can dispense and only so much power that you can bring to like cluster rack and how do you want to scale it and how do you point to scale it in multiple data centers sure you want to take it
15:09oh there hi i'm bill i one of the dojo engineers the so from a thermal standpoint and power standpoint we've designed it very modular so what you saw on the compute tile that will that will cool the entire tile so we once we hook it up to it is liquid cooled on both the top and the bottom side it doesn't need anything else and so when we talk about clicking these together once we click it to power and we once we click it to cooling it will be fully powered and fully cooled and all of that is less than a cubic foot yeah so tesla has a lot of expertise in power electronics and in in cooling so we took the power electronics expertise from the vehicle powertrain
16:03and the sort of the advanced cooling that we developed for the power electronics and for the vehicle and applied that to the super computer because as you point out getting heat out is extremely important it's just really heat limited so yeah so it's just funny that like at the compute level it's operating at less than a volt which is a very low voltage there's a lot of amps so therefore a lot of heat i squared r is what really bites you on the ass
16:38hi my question's also similarly a question of scaling so it seems like a natural consequence of using you know significantly faster training hardware is that you'd be either training models over a lot more data or you'd be training a lot more complex models which would be potentially significantly more expensive to run at inference time on the cars i guess i was wondering like if there was a plan to like also apply dojo as something that you'd be using like on the self-driving cars and if so like you know do you foresee additional challenges there i can so as you could see like andre's models are not just for cars like there are auto labeling models there are other
17:23models that are like beyond car application but they feed into the car stack so so dojo will be used for all of those too not just the car inference part of the training yeah i mean the dojo's the first application will be consuming video data for training for that would then be run in the inverse inference engine on the car but and that i think is an important test to see if it actually is good or but is it actually better than gpu cluster or not so but then beyond that it's basically a general a generalized neural net training computer but it's very much optimized to be a neural net so you know cpus and gpus
18:13they're they're they're not made to be they're not they're not designed specifically for training neural nets we've been able to make gpus especially very efficient for portraying neural nets but that's not that was never their design intent so it's it's bc gpus are still essentially running it you're on that training and emulation mode so with dojo we're saying like okay let's just let's just asic the whole thing let's just have this thing that's it's built for one purpose and that is neural net training and and just generally any system that is designed for a specific purpose will be better than one that is designed for a general purpose yeah i had a question here hi
18:58so you described two separate systems one was for vision therefore planner and control does dojo love you train networks that cross that boundary and second thing is if you were able to train such networks would you have the onboard compute capability in the fst system to be able to run that in in the under your tight latency constraints thanks yeah i think we should be able to train panel networks on dojo or any gpus it's really invariant to the platform and i think if anything once we make this entire thing end to end it'll be more efficient than decoding a lot of these intermediate states so you should be able to run faster if you make the
19:34entire thing into an neural networks we can avoid out of decoding of these intermediate states and only decode essential things required for driving the car yep certainly and to endness and as the guiding principle behind a lot of the network developments and over time in the stack neural networks have taken on more and more functionality and so we want everything to be trained end-to-end because we see that that works best but we are building it incrementally so right now the interface there is vector space and we are consuming it in the planner but nothing really fundamentally prevents you from actually taking features and eventually fine-tuning end-to-end so i think that's
20:07definitely where this is headed yeah and the discovery really is like what are the right architectures that we need to place in network blocks to make it amenable to the task so like on a describe we can place spatial rnns to help with the perception problem and now it's just neutral network so similarly for planning we need to bake in search and optimization into the planning into the network architecture and once we do that you should be able to do planning very quickly similar to c plus plus algorithms
20:40i i think i had a question very similar to what he was asking about it seems like a lot of neural nets around computer vision and kind of traditional planning you had model predictive control in solving convex optimization problems very quickly and i'd wonder if there's a compute architecture that's more suited for convex optimization or the model predictive control solutions very quickly yeah 100 if you want to bacon like i said earlier if you want to bake in these architectures that do say model protein control but just like replace some of the blocks with neural networks or if we know the physics of it we can also use physics-based models part of
21:15the neural network's forward pass itself so we are going to go towards a hybrid system where we will have neural network blocks placed together with physics based blocks and more neural networks later so it'll be a hybrid stack and what we know to do will be placed with explicitly and what the networks are created we'll use the networks to optimize this so builder end-to-end stack with this architecture baked in i i mean i do think that so as long as you've got like surround video neural nets for understanding what's going on and can convert the surround video into vector space then you basically have a video game and if you know if you yeah it's like if you're in grand theft
22:02auto whatever you can you can make the cars drive around and pedestrians walk around without crashing so you can do you don't have to have a neural net for control and planning but it's probably ultimately better so but i think you can probably get to in fact i'm sure you can get to much safer than human with control and planning primarily in c plus plus with perception vision in neural nets
22:36hi my question is we've seen other companies for example use reinforcement learning and machine learning to optimize power consumption and data centers and all kinds of other internal processes my question is are is tesla using machine learning within its manufacturing design or other engineering processes
22:59i i i discourage use of machine learning because it's really difficult unless you basically unless you have to use machine learning don't do it it's usually a red flag when somebody says we want to use machine learning to solve this task i'm like that sounds like so 99.9 percent of time you do not need it so yeah but so it's kind of like a you you reach for machine learning when you when you need to not but it's i've not found it to be a convenient easy thing to do it's a super hard thing to do that may change if you've got a humanoid robot that can you know understand normal instructions but yeah generally minimize use of machine learning in the
23:56factory hi based on your videos from the simulator it looked like a combination of graphical and neural approaches i'm curious what the set of underlying techniques that are used for your simulator and specifically for neural rendering if you can share yeah so we're doing at the bottom of the stack it's just traditional game techniques just rasterization real time you know very similar to what you'd see in like gta on top of that we're doing real-time ray tracing and then those results were really hot off the press i mean we had that little asterisk at the bottom that that was from last night we're going into the neural rendering space we're trying out a bunch of different
24:44things we want to get to the point where the neural rendering is the the cherry on the top that pushes it to the point where the models will never be able to overfit on our simulator currently we're doing things similar to photorealism enhancement there's a paper a recent paper photo enhancing photo realism enhancement but we can do a lot more than what they could do in that paper because we have way more labeled data way more compute and also much we have a lot more control over environments and we also have a lot of people who can help us make this run at real time but we're going to try whatever we can do to get to the point where we can train everything just with the simulator
25:23if we had to but we will never have to because we have so much real world data that no one else has it's just to fill in the little gaps in the real world yeah i mean the simulator is very helpful when there's like these rare cases like like you know like collision avoidance right before an accident and then ironically the better our cars become at avoiding accidents the fewer accidents there are so then our training set is small so then we have to make them crash in the simulation so it's like okay minimize potential injury to pedestrians and people in the car you have five meters you're traveling at you know 20 meters per second what actions would minimize
26:11probability of injury we can run that in some so cars driving down the wrong side of the highway that kind of thing happens occasionally but not that often for your humanoid contacts i'm wondering if you've decided on what use cases you're going to start with and what the grand challenges are in that context to make this viable well i think for the humanoid for the teslabot optimus it's basically going to start with just dealing with work that is boring repetitive and dangerous basically what is the work that people would least like to do
27:08hi so quick question about your simulations obviously they're not perfect right now so are you using any sort of domain adaptation techniques to basically bridge the gap between your simulated data and your actual real-world data because i imagine it's kind of dangerous to just deploy models which are solely trained on simulated data so maybe some sort of explicit domain adaptation or something is that going on anywhere in your pipeline so currently i mean we're producing the videos straight out of the simulator the the full clips of kinematics and everything and then we're just immediately training on them but it's not the entire data set it's just a
27:47small targeted segment and we only are evaluating based on real world video we're paying a lot of attention to make sure we don't ever fit and if we have to start doing fancier things we will but currently it's we're not having an issue with it overfitting on the simulator we will as we scale up the data and that's what we're hoping to use neural rendering to bridge that gap to push that even further out we've already done things where we're using like the same network as in the car but retrain it to detect sim versus rail to driver art decisions and that's actually helped prevent some of these things as well yeah just emphasize that overwhelmingly
28:24the data set is the real video from the cars on the actual roads nothing's weirder or or has more corner cases than reality it's gets really strange out there but but then if if we find say a few examples of something very odd and there's some very some some very hard pictures we've seen then in order to train it effectively we want to create simulations say a thousand simulations that are that are variants of that quirky thing that we saw the foot to fill in the some important gaps and and make the system better and really all of this is about over time just reducing the probability of of a crash or an injury and it's called the march of nines like how
29:12do you get to 99.999999 safe you know and it yeah each nine is an order of magnitude difficulty increase thanks so much for the presentation i was curious about the tesla bot specifically i'm wondering if there are any specific applications that you think the humanoid form factor lends itself to and then secondary because of its human form factor is emotion or companionship at all thought about on the product roadmap at all
29:58we certainly hope this does not feature in a dystopian sci-fi movie but you know like really at this point we're saying like maybe this robot can just we're trying we're trying to be as literal as possible can it do boring dangerous repetitive jobs that people don't want to do and you know once you can have it do that then maybe you can do other things too but that's the that's the thing that we really great to have so it could be your buddy too or you can buy one and have it be or your friend and whatever i'm sure that people think of some very creative uses
30:48so so so firstly thanks for the the really incredible presentation my question is on the ai side so one thing we've been seeing is that with some of these language modeling ais we've seen that scaling has just had incredible impacts in their capabilities and what they're able to do so i was wondering whether you're seeing similar kinds of effects of scaling in in your neural networks and your applications absolutely a bigger network typically we see it performs better provided you have the data to also train it with and this is also what we see for ourselves definitely in the car we have some latency consideration to be mindful of and so there we have to get creative to
31:30actually deploy much much larger networks but as we mentioned we don't only train neural networks for what goes in the car we have these auto labeling pipelines that can utilize models of arbitrary size so in fact we've traded a number of models that are not deployable that are significantly larger and work much better because we want 100 like we want much higher accuracy for the auto labeling and so we've done a lot of that and there we definitely see this trend yeah the order labeling is an extremely important part of this whole situation without the order labeling i think we would not be able to solve the self-driving problem it's kind of a funny form of
32:06distillation where you're using these very massive models plus the structure of the problem to do this reconstruction and then you distill that into neural networks that you deploy to the car but we basically have a lot of neural networks and a lot of tasks that are never intended to go into the car yeah and also as time goes on that you get new frames of information so you really want to make sure your computer is distributed across all the information as opposed to just taking a single frame and hogging on it for say 200 milliseconds you actually have newer frames coming in so you want to like use all of the information and not just use that one frame
32:38i think one of the things we're seeing is that the car's predictive ability is is quite is eerily good it's really getting better than human in terms of predicting like you say like what predict what this road will look like out when it's out of sight like it's around the bend and it predicts the road with very high accuracy and you know predict pedestrians or cyclists wherever behind you know where it just sees a little corner of the bicycle and a little bit through through the windows of the bus and its ability to predict things is going to be much better than humans like really way way beyond right yeah we see this often where we have something that is not visible but the
33:22neural network is making up stuff that actually is very sensible sometimes it's eerily good and you have to like you're wondering this is in the training set and actually actually in the limit you can imagine the neural net has enough parameters to potentially remember earth so in the limit it could actually give you the correct answer and it's kind of like an hd map back baked into the weights of the neural net
33:46okay i have a question about the design of the tesla bot specifically in order how important is it to maintain that humanoid form to build hands with five fingers that also respects the weight limits could be quite challenging you might have to use cable driven and then that also causes all kinds of issues
34:10i mean this is just going to be bought version one i mean we'll see so the it's it needs to be able to do things that that people do and you know be a generalized you know humanoid robot i mean you could make you potentially have it give it like you know two fingers and a thumb or something like that you know for now we'll we'll give it five fingers and and see see if that works out okay i probably will it doesn't need to be like you know have like incredible grip strength but it needs to be able to work with tools so and you know carry a bag that kind of thing all right thanks a lot for the presentation so an old professor of mine told me that
35:01the thing he disliked a lot about his tesla was that the autopilot ux didn't really inspire much confidence in the system especially when like objects are spinning classifications are flickering i was wondering even if you have a good self-driving system how are you working on convincing tesla owners other road users or other road users and just the general public that your system is safe and reliable well i think that's that's the cars a while back cars used to spin they don't they don't spend any more not in the if you've seen the fsd beta videos they they're they're pretty solid and they will be getting more solid yeah as you add more and more data and train
35:41these multi-camera networks like these are pretty recent actually just like few months old and they're still improving it's not a done product and that we've never minds we can clearly see how this is just going to be like perfect but perfect vector space because why not all the information is that in the videos it should produce a given lots of data and good architectures and this is just intermittent point in the timeline i mean it's clearly headed to way better than human without question
36:13my turn oh hi here i was wondering if you could talk a little bit about the short to medium-term economics of the bot i guess i understand the long-term vision of replacing physical labor but i also think repetitive dangerous and boring tasks tend to not be so highly compensated and so i just don't see how to reproduce you know start with a supercar and then break into like the lower end of the market how do you do that for a robot humanoid well i guess you'll just have to see
36:55hello hi i was curious to know how the car ai prioritizes occupant safety versus pedestrian safety and what thought process goes into like deciding how to make this into the ai well i mean we the the thing to appreciate is that from the computer standpoint everything is moving slowly i think you know to a human things are moving fast to the computer they are not moving fast so i think this is in reality somewhat of a false dichotomy not that it will never happen but it will be very rare you know if you think it was like you know going the other direction like rendering you know with full ray tracing neural net enhanced graphics on something like cyberpunk or in any you
37:49know advanced video game you know doing 60 frames a second perfectly rendered like how long would it take a person to even render one frame and without any mistakes can't be done i mean it would take like a month just to just render one one frame out of 60 in a second in a video game it's a computer are fast and humans are slow
38:23i mean for example on on the rocket side the you you cannot steer the rocket to orbit we actually hooked up a joystick to see if anyone could steal the rocket orbit but you need to react at roughly six seven hertz people can't do it not even now that's pretty low you know we're talking more like aiming for like 30 hertz type of thing
38:55hi with the over here with hardware 3 there's been lots of speculation that with larger nets it's hitting the limits of what it can provide how much headroom has the extended compute modes provided and what point would hardware 4 be required if at all well i'm i'm confident that hardware 3 or the full stop driving computer one will be able to achieve full self driving at a safety level much greater than a human probably i don't know at least two or three hundred percent better than a human then obviously there will be a future hardware for or full self-driving computer too which we'll probably introduce with the cybertruck so maybe in about a year or so
39:42that it probably will be about four times more capable roughly but it's really just going to be like can we take it from say for argument's sake 300 safer than a person to a thousand percent safer you know just like there are people on the road who with with varying driving abilities but we still let people drive that you don't have to be the world's best driver to be on the road so as we see so yeah guess what's neat
40:21so are you worried at all since you don't have any depth sensors on the car that people might try like adversarial attacks like printed out photos or something to try to trick the rgb neural network yeah like pull some like wiley cody stuff you know like paint the tunnel on the on the wall it's like oops we haven't really seen much of that i mean for sure like like right now if you pro most likely if you had like a a t-shirt with a t-shirt with like a stop sign on it which i actually have a t-shirt with a stop sign on it and and then you like flash the car it will it will stop i i proved that but we can obviously as we see these adversarial attacks then we can we train
41:19the cars to you know notice that well it's actually a person wearing a t-shirt the stop sign on it so it's probably not a real stop sign hi my question is about the prediction and the planning i'm curious how do you incorporate uncertainty into your you know planning algorithms do you just basically assume you know you mentioned that you run the autopilot for all the other cars on the road do you assume that they're all going to follow those rules or are you accounting for the possibility that well they might be bad drivers for example yeah we do account for multi-modal futures it's not that we just choose one we account for this person can actually
42:06do many things and we use that actual physics and kinematics to make sure that they're not doing a thing that would interfere with us before we act so if there's any uncertainty we are conservative and then would yield to them of course there's a limit to this because if you're too conservative then it's probably not practical so at some point we have to assert and we even then we make sure that the other person can yield to us and act sensibly
42:34i should say like like before we introduce something into the fleet we will run it in shadow mode and so and we'll see what what would this neural net for example have done in this particular situation because and then effectively the drivers are training it training the net so if the neural net would have controlled and you know and say veered right but the person actually went left it's like oh there's a difference why was there that difference yeah and secondly all the human drivers are essentially training the neural net as to what is the correct course of action assuming it doesn't then ended up in a crash you know doesn't count in that
43:19case yeah and secondly we have various estimates of uncertainty like flicker and when we observe this we actually say we are not able to see something we actually slow down the car to be again safe and get more information before acting yeah we don't want to be brazen and just go into something that we don't know about we only go into places where we know about yeah yeah so it should be like aspirationally that the car should be the less it knows the sl you know the slower it goes yeah which is not true at some point but now yeah yeah we've yeah should we speed proportionate to confidence
44:00i'm sorry thanks for the presentation so i am curious appreciate the fact that the fsd is improving but if you have the ability to improve one component along the ai stack that presented today whether it is simulation data collections planning control etc which one in your opinion is going to have the biggest impact for the performance of the full self driving system
44:31it's really the area under the curve of this like multiple points and if you improve anything it should improve the area i mean the short term it's arguably we need all of the nets to be surround video and so we still have some legacy this is a very short term obviously we're fixing it fast but there's there's still some nets that are not using surround video and i think ideally that all use surround video yeah very yeah i think a lot of puzzle pieces are there for success we just need more strong people to also just help us make it work yeah that is the actual box so that is the actual bottleneck i would say i'm really one of the reasons that
45:09we are putting on this event exactly what well said andre that there's just a tremendous amount of work to do to make make it work so that's why we need talent people to join and solve the problem thank you for the great presentation lots of my questions answered but one thing is when imagine that now you have a large amount of data even unnecessary how do you consider that like there's a forgetting problem in neural networks like how are you considering those aspects and also another one are you considering online learning or continuous learning so that maybe each driver can have their version of self-driving software i think i think i know the literature
46:06that you're referring to that's not some of the problems that we've seen and we haven't done too much continuous learning we trained the system once we fine-tuned a few times that sort of goes into the car we need something stable that we can evaluate extensively and then we think that that's good and that goes into cars so we don't do too much learning on spot or continuous learning and don't face the forgetting problem but there will be settings that you can say like if you do do you want are you typically a conservative driver or do you want to drive fast or slow you know it's like i'm late for my i'm late for the airport could you go faster than
46:37you know basically the kind of instructions you'd give to your uber driver it's like i'm late for the flight please hurry or take it easy or you know whatever your style is so let's take a few more questions here so and then we'll call it a day all right so as our models have become more and more capable and i guess you're deploying these models into the real world one thing i guess that's possible is for ai to become more i guess misaligned with what humans desire so i guess is this something that you guys are worried about as you guys deploy more and more robots or do you guys like we'll solve that problem when we get there yeah i think that we should be worried
47:21about ai now like what we're trying to do here is i say a narrow ai pretty narrow like just make the car drive better than a human and then have the humanoid robot be able to do basic stuff you know so at the point which you sort of start get superhuman intelligence yeah i don't know all bets are off but you know and that's that's that's you know that'll that'll probably happen but but what what we're trying to do here at tesla is make useful ai that people love and and is unequivocally good that's our you know try to aim for that okay maybe one more question
48:15hi my question is about the camera sensor in the beginning of the talk you had mentioned about building a synthetic animal and if you think about it a camera is a very poor approximation of a human eye and a human eye does a lot more than take a sequence of frames have you looked into like like these days are like cameras like event cameras have you looked into them or are you looking into a more flexible camera design or building your own camera for example well with hardware four we will we will have a next generation camera but i have to say that the the current cameras we have not reached the limit of the current cameras so and i'm confident we can achieve full
48:59self driving with much higher safety than humans with the current cameras and current compute hardware but you know are we good to be a thousand percent better rather than 300 better so we'll see continued evolution on on all levels and pursue that goal and i think in the future people will look back and say wow i can't believe we had to drive these cars ourselves you know it's like self-driving cars will just be just a normal like self-driving elevators you know elevators used to have elevator operators and there's someone there with like you know big big relay switch operating the elevator and then every now and then they'd get tired or you know some make a mistake
49:43and share somebody in half so so now we you know we made elevators automatic and you just go and you press the button and you can be in a 100 story skyscraper and don't really worry about it just go ahead and press a button and the elevator takes you where you want to go but it used to be that all elevators will operate manually manually it'll be the same thing like for cars all cars will be automatic and then and electric obviously so there will still be some gasoline cars and some manual cars just like there are still some horses so all right well thanks everyone for coming and i hope you enjoyed presentation and thank you for the great questions [Applause]
资源与出处
- 视频在 YouTube 收看
- 档案silicon-valley.tech 档案页本页视频与文稿数据来自 Elon Musk Archive
- 精读本篇精读 Markdown 版