发布会·中文精读

Tesla AI Day 2021 全程回顾

Tesla AI Day 2021

B 站版本为网友搬运,时间点可能差几秒
中文精读由 AI 基于原始视频与文稿改写,非官方翻译,可能存在偏差或错误;表述以原视频为准。说明

一辆半挂车在高速上行驶,特斯拉车身上的 8 个摄像头里,它可能同时出现在其中 5 个画面中。每个摄像头只能拍到它的一部分,车头、车厢中段、车尾、侧面,或者后视镜里的一角。

2021 年的 Tesla AI Day 上,视觉团队负责人 Andrej 用这个场景解释了一件事:整套感知架构必须推倒重来。

在那之前,特斯拉的感知是逐摄像头的。每个神经网络只看自己那张图,输出预测,再由一个 C++ 模块把各摄像头的预测缝合起来,跨越摄像头边界,按时间叠加。

智能召唤功能很快把这条路堵死了。5 个独立网络各自猜测这辆半挂车的尺寸、朝向和位置,再想办法融合成一个结果。融合出来的东西没法看。

更麻烦的是投影。车道线和路缘在原始图像上画得很准,一投影到鸟瞰图就扭成一团。要让这个投影成立,每个像素的深度都得准到离谱的程度。被遮挡的地方更麻烦,图像空间里根本没有对应的像素,网络无从预测。

马斯克上台时的第一句话定了个调:特斯拉远不只是一家电动车公司。接下来三个多小时,视觉、规划、标注、仿真、芯片、超算六个团队轮流上台,把这句话拆成了具体的工程量。

从一只人造动物说起

Andrej 描述他们在做的事:从零开始造一只合成的动物。

车身是躯体,线束是神经系统,Autopilot 是大脑,视觉网络是皮层。他说得很认真:光线打到视网膜,经过 LGN,传到 V1、V2、V4,分成腹侧流和背侧流,信息是按特定布局组织的。设计车的视觉皮层时,要决定的其实是同一件事,信息在网络里怎么流。

四年前的网络长这样。1280×960 的输入,12 位整数,每秒 36 帧,先过一个 RegNet 主干,输出一组多尺度特征。最底层分辨率高、通道少,神经元盯着很小的图像块;最顶层分辨率低、通道多,每个神经元能看到大半幅图。

中间接特征金字塔,让不同尺度互相通信。Andrej 举的例子很形象:底层有个神经元盯着一个小块,不确定眼前是不是车。如果顶层告诉它“你正处在高速公路的消失点附近”,判断一下子就容易了。

金字塔之后接任务头。检测头是 YOLO 式的一阶段结构,初始化一个栅格,每个位置一个二进制位表示有没有车,再挂上 x、y、高度偏移、车型这些属性。

问题很快冒出来:要做的任务远不止检测车。红绿灯、车道线、路缘、锥桶、垃圾桶,全都要。于是他们改成共享主干加多分支,起了个名字叫 HydraNet,九头蛇网络。一个身体,好几条脖子。

这个结构带来三个实打实的好处。推理成本被摊薄,车上跑一次主干,所有任务共享。任务之间解耦,改某个头的数据集或者结构,不需要把其他任务全部重新验证一遍。特征可以缓存,多尺度特征存到磁盘,微调时只从缓存特征往上训练头。他们的工作流是:端到端训一轮,缓存,微调一阵,再端到端训一轮。

但这套东西在智能召唤上线时碎了。

Transformer 把像素搬进三维空间

先是调参的问题。那个负责缝合摄像头边界和时间的 C++ 模块叫 occupancy tracker,超参数极多,手调极其痛苦。工程师真正想要的是把这些东西全部塞进神经网络里,端到端训练。

再是输出空间的问题。图像空间本来就不是正确的输出空间。Andrej 放了一组对比图,路缘和车道线在原始图像里画得严丝合缝,投到向量空间立刻变成一团废线。

他的结论很直接:所有图像同时喂进一个网络,直接输出向量空间。

说起来一句话,做起来是两座大山。第一座:怎么造出能完成这个变换、而且可微的神经网络组件,让端到端训练继续成立。第二座:向量空间的预测需要向量空间的标注,光给图像打标签是不够的。

第一座山的解法是 Transformer 加多头自注意力。

流程是这样。先初始化一个和目标输出空间同样大小的栅格,用 sin/cos 位置编码铺满,再通过一个 MLP 编码成一组 query 向量。同时,所有图像的特征发出各自的 key 和 value。query、key、value 一起进多头注意力。

Andrej 的解释很通俗。每个图像块在 key 里广播的是“我是这个位置一根柱子的一部分,我看到了这些东西”。每个 query 说的是“我是输出空间这个位置的像素,我在找这类特征”。key 和 query 相乘交互,value 就被相应地拉过来。

实际效果是,一个 Transformer 块就能承担大部分工作。

视频里放了对比。左边是旧方案的输出,右边是多摄像头网络直接从神经网络里预测出的向量空间,效果基本是昼夜之差。这个结果已经能直接在上面开车了。

目标检测也跟着受益。两个车在很近的距离擦身而过,摄像头边界处的预测会剧烈抖动,大车尤其明显。多摄像头网络在这些场景里稳定得多。还有那种只露出一条缝的车,单摄像头根本没法预测位置,多摄像头网络没有这个问题。

每辆车都“有点歪”

工程上还有一个容易被忽略的坑:所有车的摄像头安装位置都有细微差异。做图像空间到输出空间的变换时,摄像头标定必须准确,而且必须想办法喂给网络。

朴素的做法是把 8 个摄像头的标定参数拼起来,用 MLP 编码进网络。他们试了,但效果不是最优。

更好的做法是加一个整流的虚拟相机层。这个层是相机标定的函数,把 8 张图全部转换成一个合成虚拟相机的视角。效果可以这样看:把后向 repeater 相机的大量图像取平均,不做整流会得到一张糊掉的后视镜,做了整流之后,后视镜边缘变得非常锐利。

这个改动不算大,性能提升相当可观。

给网络装一段记忆

多摄像头解决了空间问题,时间问题还悬着。

有一大批预测必须要视频上下文才能做。这辆车是停着还是在动,速度多少,刚才被挡住了现在还在不在,前方道路的几何形状。Andrej 说得很具体:要预测前方路口的车道走向,50 米之前看到的那块路牌和标线非常关键。

他们在多尺度特征和任务头之间插了两个模块:feature queue 和 video module。

feature queue 负责把特征随时间缓存下来,同时把车辆的 kinematics 也编码进去。这里有个细节值得说:他们做了两种队列。

时间基准队列,比如每 27 毫秒进一帧。车被临时遮挡,网络可以回头看记忆,知道刚才那里有东西。

空间基准队列,车每行驶固定距离进一次。为什么需要这个?假设你在等红灯,路口的车道线是 30 秒之前看到的,纯时间队列早就把它忘了。空间队列能保证关键信息不会因为等待而丢失。

video module 的选择花了点功夫。3D 卷积、Transformer、轴向 Transformer、各种变体的循环神经网络,都试过。最后他们比较中意的是空间循环神经网络。

设计的出发点是:车在二维路面上跑,那就把 hidden state 组织成一个二维点阵。车在路上移动时,只更新离车近的、有可见性的那一块。位置通过 kinematics 积分进网格。

训练完之后有个细节挺有意思。点阵里不同的通道自己学会了追踪不同的东西,有的追踪道路中心,有的追踪边缘,有的追踪路面材质。

Andrej 说,这相当于一张画在特征空间里的 HD 地图。区别在于它的组成单位:显式的路段和标线条目被 RNN 的隐状态取代了。多个车、多次经过同一地点的 clip 可以协作,共同把这张图建出来。

视频记忆对目标检测的提升也很直观。两个车开过来挡住前方停着的车,单帧网络会直接丢掉目标,视频模块能把它们保持住。只露出一部分的时候,单帧网络被迫给一个很烂的猜测,视频模块知道这一刻信息不足,干脆不采信。

移除雷达的那一轮,视频模块的速度估计几乎追平了雷达信号。单帧网络靠深度差分算速度,误差明显更大。

完整架构堆起来是这样的:原始图像,整流层,RegNet 主干,特征金字塔,Transformer 转成向量空间,feature queue,空间 RNN,HydraNet 的各种任务头。

Andrej 提了两个正在做的改进:融合时机偏晚,能不能在更靠底部的层做代价体积或者光流;输出是稠密栅格,后处理在车上的开销很大,他们在研究直接输出稀疏结构,比如逐点预测。

规划:非凸和高维的两难

拿到向量空间之后,规划控制组接手。负责人 Ashok 说,规划有两个本质难点。

第一,动作空间非凸。可能有多个分别不错的解,但找到一个全局一致的解非常难,规划器容易卡在某个局部最优里出不来。

第二,维度极高。车要规划未来 10 到 15 秒的位置、速度、加速度,运行时需要生成的参数量非常大。

这两件事把两类经典方法各自逼进了死角。离散搜索不怕非凸,因为它不靠梯度,不会卡在局部最优,但它是离散的,用不上梯度信息,只能逐个点去探索,维度一高就崩。连续优化反过来,梯度一导很快就到好解,但非常容易陷进局部极小。

他们的解法是分层。先用离散搜索把非凸性压下去,搜出一个凸走廊,再在这个走廊里做连续优化,生成平滑轨迹。

举例:车要在路口前连续变两次道,然后左转。搜索会试上千种操作。第一次试的是就近变道,车会急刹,很不舒服;第二次试稍微晚一点变,加速超过旁边的车,在它前面变道,但这样可能赶不上左转。上千次这样的搜索在极短时间内跑完,最后按安全、舒适、能顺利完成左转这三个条件挑一个。

实际执行时,规划轨迹和真实速度曲线基本吻合,10 秒的规划能对上。

更关键的一点是联合规划。路上的其他车不能当成静态障碍物,要跑同一套 planner 在它们身上。

视频里有一段窄道会车的场景。自动驾驶绕开路边的停车和锥桶往前走,迎面来一辆车。自动驾驶先减速,然后判断自己这边没有避让空间,对方可以靠边,于是有底气地继续往前。第二辆来车速度更高,自动驾驶给这辆车也跑了一遍 planner,算出它的方案是绕过它那一侧的停车再回到自己车道。同时算法维护多个可能的意图,绿色的是对方让行,红色的是对方强行通过。因为对方速度和加速度都很高,系统判断它多半会走红色那条,于是选择先靠边。靠边的过程中观察到对方的横摆角和加速度变化,确认对方选择了让行,立刻改主意继续走。

Ashok 的解释很到位:如果不对其他车辆做规划,自动驾驶会过于胆小,根本没法当一辆实用的车。

最后一步是连续优化。以凸走廊为约束,用弧长参数化的样条曲线作为初值,不断微调降低总成本。成本项包括离障碍物的距离、通过时间、舒适度。舒适度可以从右侧的图上看出来,横向加速度是漂亮的梯形,和人类驾驶员录下来的曲线很接近,横向 jerk 也被压到很低。

288 个节点,对比 40 万个节点

人工设计启发式函数这条路是有天花板的。Ashok 用一个停车问题把差距量化了。

蓝色是本车,绿色是目标车位,橙色是路缘、停车和锥桶。

标准的 A* 加欧氏距离启发,车会径直朝目标冲,很快撞进局部最优,然后回溯,再去试别的路径。最终能到,但用了 40 万个节点。

加上导航路线,让搜索沿着导航走、同时靠近目标,好一些,但遇到锥桶还是得回溯。2.2 万个节点。

继续设计更精细的启发式当然可以,但这是无底洞,而且很难设计出全局最优的那一个。

换成神经网络提供价值函数,再接上蒙特卡洛树搜索。神经网络能吸收整个场景的全局上下文,直接给出一个有效的价值估计,引导搜索走向全局最优。同一个停车问题,288 个节点。差了三个数量级。

Ashok 把这个问题比成多人版雅达利游戏。视觉网络把稠密视频压成向量空间,车和行人都在里面移动,这本身就是一个游戏状态。AlphaZero 和 MuZero 那一套方法可以直接搬过来。

最终架构是显式规划和神经网络规划并行。视觉系统输出的向量空间同时喂给两边,神经网络规划器还可以吃网络的中间特征。两边一起产生轨迹分布,训练信号混合显式成本函数(距离、碰撞、舒适、通行时间)和人类驾驶干预数据。轨迹分布再交给显式规划器做它擅长的部分,最终输出转向和加速度指令。

一千人的标注队伍

回到 Andrej 的话:神经网络只决定性能的上限。几亿个参数,如果设置不对就是一堆废物。真正把这些参数训练到正确位置的是数据。

而他们需要的,是直接落在向量空间里的数据。

四年前,数据标注是外包给第三方的。很快发现这条路走不通。关键任务的外包延迟极高,质量也说不上好。于是按特斯拉一贯的垂直整合思路,全部收回自建,长成了一支一千多人的专业标注团队,和工程师在同一片办公区办公,标注基础设施也是自己从零搭的。

早期的标注在图像空间里做,画多边形和折线,效率极低。后来升级到三维、四维标注,直接在向量空间里改,改完再重投影回各个相机图像。吞吐量涨了一大截。

但他们很快意识到,光靠人也不够。人和计算机各有所长:人非常擅长语义判断,计算机非常擅长几何重建、三角化、跨帧追踪。真正的答案是把两者放进同一条流水线里。

服务器可以“作弊”

Ashok 接着讲自动标注。一个 clip 是一段 45 秒到一分钟的密集传感器数据,视频、IMU、GPS、里程计全都包含,来源可以是工程车也可以是用户车。这些 clip 上传到服务器,先跑一大堆离线神经网络,得到分割掩码、深度、点匹配等中间结果,再经过一整套机器人算法,产出最终标签。

路面重建用了隐式表示。传统的样条或者网格受拓扑限制,不可微,不好优化。他们参考了 NeRF 的做法,在地面上采样 xy,让网络预测这个位置的高度 z 和语义,比如这里是不是车道线、路缘、路面。一个 xy 加上预测的 z 构成一个三维点,投影回 8 个摄像头,和图像空间的分割预测对比。数百万次采样,跨相机、跨时间联合优化。

一个车开过一段路,只能扫出轨迹附近的一小片。他们就把 16 段不同行程的 clip 放进一个巨大的优化里,用车道线和路缘这类特征对齐,让它们互相吻合,同时和各自的图像观测吻合。这样标注不局限于车开过的地方,周边也能覆盖到。

静态障碍物的三维重建也有个亮点:不需要纹理。传统上跨帧匹配要依赖纹理做关联,在路面和墙面这种弱纹理区域很难做。他们的方法能在这些地方也产出点云,对任意障碍物的标注都很有用。规划这边离不开这个东西,因为只要曾经见过一次,就必须一直把那个人记在账上。

离线处理有一个车上永远拿不到的优势:能看见未来。

在车上,网络只能用历史推断速度。在服务器上,历史帧和未来帧都在手里,可以直接“作弊”,给出精确的 kinematics。遮挡后的轨迹关联也变得简单:知道未来,就能把断掉的轨迹接回去。视频里几个行人在多辆车遮挡之后依然被正确关联。

目标是产出一百万个这样标注的 clip。

移除雷达项目是最好的案例。三个月内完成。早期网络在恶劣天气下会失效,比如前面一辆卡车扬起大片雪雾,几乎看不见,但逻辑上该记得前面有辆车。车队被派去收集类似场景,自动标注流水线一周处理了 1 万个 clip,同样工作量交给人工要几个月。他们在 200 种不同的恶劣条件下做了同样的事。

训练完之后,雪雾里那辆车被记住了。

在仿真里造一台 Cybertruck

Ashok 在台上放了一段 Cybertruck 跟着车流走的片段,问观众:这段视频哪来的?有人猜对了,是渲染的。

细节做得非常足。地面上有裂缝、有接缝、还有后期补上的沥青补丁,看着就很旧;卡车是铰接的,转弯时后轮会压过路缘;其他车行为合理,会避让、会绕行、加速平滑、刹车平滑;画面上方那辆带标志的车是 Autopilot 自己在开,正在做一个不保护的左转。

仿真从向量空间起步,所以标签是完美的。他能列出一堆现成的标注:车辆的三维框加运动学、深度、表面法线、分割。Andrej 下周想要一个新任务,代码写起来很快,因为底层的向量空间已经在了。

仿真什么时候值得用?Ashok 给了三个判据。

数据难获取。再大的车队也收集不到某些极端场景,比如一对夫妻带着狗在高速上跑,旁边还有其他高速行驶的车。

数据难标注。几百个行人同时过马路,人工标注要几小时,自动标注算法也很难把关联做对、速度算准。仿真里这些都是现成的。

闭环行为。数据本身取决于车的动作,这种只有仿真能给。

要让这一切成立,传感器仿真必须精确。摄像头的噪声、运动模糊、光学畸变、车灯透射、挡风玻璃的衍射花纹,全都要建模。这些模型不只用在软件上,也直接用于硬件决策,比如镜头设计、相机设计、传感器布置。

渲染要真实,游戏行业说的“锯齿”是致命的破绽,所以要花大力气做时域抗锯齿,同时上光线追踪做全局光照。资产库有几千个模型,能换衣服,动作合理。已经建了 2000 英里道路,差不多是美东到美西的距离,单个美术一天能再造几英里。但大部分训练数据其实是程序化生成的,靠曲率、树木、锥桶、车辆速度和交互的参数组合,源源不断地产出。

更聪明的一步是用机器学习找网络的失败点,在失败点周围生成更多数据,形成闭环。

他们还做了失败回放。一个真实 clip 经过自动标注,重建出三维场景和所有运动物体,加上原始视觉信息,合成出一模一样的仿真环境。让 Autopilot 在这里面重跑,可以跑出完全不同的结果,把一次失败变成一个训练样本。再叠加神经渲染,画面几乎和真实相机拍的一模一样。

到 AI Day 为止,训练已经用掉 3 亿张图片、接近 5 亿个标签。

Dojo:一块芯片,100% 给机器学习

Milan 先给了一个数量感。移除雷达那一轮,覆盖 250 万个 clip,生成了超过 100 亿个标签,动用了数千块 GPU、接近 2 万台 CPU,还把 2000 多台真实的 FSD 计算平台接入仿真环路。他说这是他手上最小的一个计算集群。

上车有两个硬约束:延迟和帧率。这关系到安全,也关系到对周围加速度和速度的准确估计。核心难点在 AI 编译器,把 PyTorch 模型的计算图映射到专用加速硬件上,同时排出一个吞吐最优的调度,还要在极其紧张的 SRAM 预算下工作。FSD 计算机上有两个引擎,任何时刻只有一个向车辆输出控制指令,另一个当算力扩展用,硬件层面和软件层面都可以互换角色。

评估规模是每周超过 100 万次,跑在 3000 多台 FSD 计算平台组成的专用集群上。训练算力接近 1 万块 GPU,比全球排名前五的公开超算加起来还多。

然后 Ganesh 上台讲 Dojo。

他的开场很直接:训练算力的需求是无止境的。几年前马斯克让他们造一台超快的训练计算机,于是有了这个项目。目标是 AI 训练性能最强,同时能效和成本都可控。

架构选了分布式二维网格。所有训练机器本质上都是分布式的,算力容易扩,带宽极难扩,延迟几乎扩不动。所以设计从延迟和带宽出发,反推节点大小。

节点大小的选择很讲究。太小,同步开销会盖过计算;太大,硬件实现复杂,最终撞上内存带宽瓶颈。他们的做法是:取信号在 2GHz 以上时钟下能跑过的最远距离,画一个框,用线把这个框塞满,这代表能喂进去的最高带宽,然后在下面放机器学习计算单元、一大块 SRAM,再加一个可编程核心负责控制。最小的训练节点就这么定下来了。

这个节点是一个 64 位超标量 CPU,围绕矩阵乘单元和向量单元优化,支持 FP32、BF16 和一个新的可配置 FP8 格式,带 1.25MB 的 ECC 保护 SRAM,单节点算力超过 1 TFLOP。它有四个硬件线程,自定义指令集为机器学习负载做了优化,转置、gather、链路遍历、广播这些都是原生指令。

354 个这样的节点拼成一个计算阵列。阵列周围围了 576 个高速低功耗串行解串器,芯片对外的 IO 带宽是当下最先进网络交换芯片的两倍多。网络交换芯片通常被认为是 IO 带宽的金标准。

这就是 D1 芯片。7nm 工艺,500 亿晶体管,645 平方毫米。整个芯片面积 100% 用于机器学习训练和带宽,没有暗硅,没有历史包袱式的兼容逻辑。从架构到 GDS 到封装,全部由特斯拉内部完成。

Ganesh 给了它一个定位:GPU 级别的算力,CPU 级别的灵活性,两倍于网络交换芯片的 IO 带宽。

九个 petaflop 拿在手里

1500 个 D1 无缝连起来,就是计算平面,包含 50 万个训练节点。两端接上 Dojo 接口处理器,一边是 PCIe Gen4 连主机,一边是高带宽 fabric 连计算平面,同时提供共享的高带宽 DRAM 和更高基数的网络连接。

系统的基本单位是训练瓦片。做法是把 25 颗经过测试的 D1 裸片集成到一片扇出晶圆上,保持裸片之间的带宽不被削减,再配一个高带宽高密度的连接器,把带宽原封不动地引出来。

Ganesh 说这大概是芯片行业里最大的有机多芯片模块。设计过程非常痛苦,因为工具根本不存在,连他们的计算集群都跑不动。工程师自己造了新方法才做出来。

供电也重新做了。他们做了一个定制的电压调节模块,可以像贴片元件一样直接回流焊到扇出晶圆上,再把 PCB 级的回流工艺和扇出晶圆工艺接起来。电气、散热、机械全部集成在一个瓦片里,52V 直流输入。

整个计算平面与供电、散热完全正交,这是让高带宽计算平面成立的关键。

一块瓦片 9 PFLOPs。Ganesh 举着它说了一句:真不敢相信我手里拿着九个 petaflop。就在上周,第一块能工作的训练瓦片在有限的台式散热条件下跑起了网络。他还补了一句,Andrej 说过,除非能跑起他自己做的东西,否则他不信。于是他们跑了 minGPT-2。

往上堆:2×3 块瓦片一个托盘,两个托盘一个机柜,100 PFLOPs。十个机柜拼成一个 ExaPOD,1.1 EFLOPS,超过 100 万个训练节点,瓦片之间带宽无损,没有衰减也没有截断。

软件栈是另一条腿。编译器负责提取细粒度并行,做数据并行、模型并行、图并行,还会做降低内存占用的优化。高带宽 fabric 让模型并行可以突破单芯片边界,扩展到训练瓦片甚至更大范围,这意味着大模型可以在低 batch size 下高效映射。编译器还能处理循环、if-else 这类动态控制流,PyTorch 扩展保持 ML 科学家熟悉的接口,LLVM 后端生成硬件二进制,不需要一行手写 kernel。

对比数据:同等成本下 4 倍性能,每瓦性能 1.3 倍,占地小 5 倍。

马斯克后来给了一个很清晰的验收标准。

Dojo 成功的标志是,软件团队主动想关掉 GPU 集群。如果他们还想继续开 GPU,那就不算成功。(马斯克)

Tesla Bot 和劳动力无限的经济学

最后一段是马斯克自己讲人形机器人。

他的论证起点是:特斯拉大概是世界上最大的机器人公司,因为车本身就是轮子上的半感知机器人。既然有 FSD 计算机当推理引擎,有 Dojo 当训练机器,有一堆神经网络在理解世界、在世界里导航,那把这一整套放进一个人形躯壳里就很自然。电池、传感器、执行器,特斯拉也都熟。

明年可能会有原型。身高 5 尺 8 寸,头部是一块屏幕,装了 8 个摄像头和 FSD 计算机,用和车上完全一样的工具链。他讲了一句让人笑了的话:设计上你可以跑得比它快,机械层面大概率也能制服它。希望永远不会用到。

定位是替代危险、重复、无聊的工作。他给的场景是:你指着地上一个螺栓说,把它拿起来,用那把扳手装到车上,它应该能完成。你说去商店帮我买这些东西,它也应该能去。

然后是那段关于经济的论述。

如果你问经济是什么,它的底层是劳动力。资本设备本质上只是蒸馏过的劳动力。那么当劳动力不再稀缺,经济还有什么真正的上限吗?(马斯克)

他的推论是长期需要某种形式的全民基本收入。但在机器人真正能干活之前,这件事不用急。再往后,体力劳动会变成一种选择,想做就做,不做也能活。

问答:从 HW4 到 AI 对齐

现场提问持续了一个多小时,信息量不小。

关于开源。马斯克说造这套系统极其昂贵,全开源的话不知道谁来付钱。但其他车厂如果想授权使用,特斯拉是欢迎的,这套东西从一开始就不打算只服务于特斯拉自己的车。

关于 Dojo 编译器是不是已经解决了扩展问题。Ganesh 答得很实在:还没解决,但有信心解决。原型硬件上已经跑起了网络,性能模型显示了扩展路径。难点正是提问者说的局部性,只要模型并行和数据并行能做到足够多,把大部分通信限制在局部,参数能装进 SRAM 工作集,就能继续扩。

关于仿真会不会扩展到公司其他部门。仿真负责人 Ian 说,走向全自动驾驶的过程中,车辆动力学、电池管理、整车控制器都要进仿真,这会让这套平台在 Autopilot 之外也有用,最终目标是通用仿真平台。

关于不同国家怎么迁移。马斯克说训练数据来自大约 50 个国家,但要解决难题,得先挑一个国家打透,因为特斯拉在美国,就先解决美国。加拿大的路看着像,其实差别够大,现在不想额外增加复杂度。Andrej 补充说计算机视觉组件本身对国家不敏感,路缘、车、道路这些概念是通用的。马斯克加了一句最高指令:别撞上。哪怕高速上掉下来一个 UFO,也不需要认识它才能不撞它。

关于几何精度。有人问多视角重建的误差有多大,答:标准差通常在 3 到 4 厘米。马斯克说目标是厘米级的定位精度,近处更准,远处可以松一些,因为远处不需要那么早做决策。

关于一体压铸和制造。有人问“外星无畏舰”那套全自动化制造理念是不是被放弃了。马斯克说有些环节几乎全自动,有些环节几乎全手动,走完整条产线会看到非常宽的分布。但绝大部分已经自动化了。

他顺势讲了车身工艺的变化。改成大型铝高压压铸之后,整个后三分之一车身可以一次铸成一个零件,前三分之一也这么做。车身产线的尺寸因此缩小了 60% 到 70%。

有人把这理解成机器人在替代产线工人,马斯克纠正了:Optimus 不是被制造需求催出来的。逻辑是反的。反正要造人形机器人需要的那些零件,那就干脆造一个。而且如果特斯拉不造,别人也会造,所以还是自己造,并且确保它安全。

他补了一句关于制造的判断:量产制造极其困难,而且被严重低估。这话对 Optimus 同样成立,如果造出来贵得离谱,那它就没有意义。

一立方英尺里的九个 petaflop

关于散热和供电,Dojo 工程师 Bill 给了答案。整个瓦片上下两面都是液冷,接上电、接上冷却,就同时完成通电和散热,整套东西体积不到一立方英尺。

马斯克补充了这套能力的来源。特斯拉在电力电子和散热上有大量积累,车用动力总成的电力电子技术、为功率器件开发的先进散热方案,直接搬到了超算上。他讲了个容易被忽略的细节:计算芯片工作在不到 1 伏的电压下,电压低意味着电流大,电流大意味着发热多,真正咬人的是 I²R 这一项。

有提问者关心 Dojo 会不会下放到车端推理。Andrej 答得很清楚:Dojo 会服务于所有模型,包括大量根本不进车的模型,比如自动标注、仿真里的那些。这些模型的输出最终会喂养车上的推理引擎。Dojo 的第一个应用是消化视频数据做训练,这是它最合适的场景。

马斯克从更抽象的层面解释了为什么要自己造芯片。GPU 从来不是为训练神经网络设计的,只是恰好被用得比较高效,本质上仍在用模拟的方式跑神经网络训练。Dojo 的做法是把整件事 ASIC 化,只为一个目的建造。为特定目的设计的系统,一定优于为通用目的设计的系统。

端到端是一条路,不是一步

有人问,视觉和规划现在分成两个系统,Dojo 能不能训练跨越这条边界的网络,车上的算力够不够跑。

Andrej 说能,而且训练平台本身无关紧要。他还提到一个和直觉相反的可能:端到端之后速度反而更快。现在要在中间解码大量稠密栅格,很多计算是浪费的。端到端可以直接跳过这些中间表示,只解码开车真正必需的东西。

Ashok 补充了方法论。端到端一直是指导原则,现在两边的接口是向量空间,但没有什么根本性的东西阻止把特征接过去做联合微调。真正需要想清楚的是:哪些模块该被放进网络架构里。

他举了自己这边的例子。感知这一侧,他们把空间 RNN 放了进去。规划这一侧,同样要把搜索和优化烘焙进网络结构,这样才能像 C++ 算法一样快速规划。

被问到有没有更适合凸优化或者模型预测控制的硬件架构时,Ashok 给出了混合系统的判断:把神经网络块和基于物理的块放在一起,已经知道怎么精确算的部分就用显式方法,剩下的交给网络去优化。这就是他们要走的方向。

马斯克给了另一个视角。只要环视视频的神经网络足够好,能把画面转成向量空间,你手上就等于有了一个电子游戏。他说得很形象,就像《侠盗猎车手》里那样,你可以让车流走起来、行人绕开障碍不撞上。控制和规划不一定非得用神经网络。但长远看,用了会更好。他也确认了自己的判断:靠 C++ 做控制和规划、靠神经网络做感知,已经能做到比人类安全得多。

“除非你不得不用机器学习,否则别用”

有提问者问特斯拉在制造和工程流程里有没有用机器学习。

马斯克的回答值得原样记下来:

我劝大家少用机器学习,因为它真的很难。除非你不得不用,否则别用。有人跑来说“我们想用机器学习解决这个问题”,这通常是个危险信号。99.9% 的情况下你根本不需要它。(马斯克)

他承认这个判断可能会被人形机器人改变,因为那类需要理解自然语言指令的场景确实绕不开。但在工厂里,他的态度是最小化使用。

这段和整场发布会的基调构成一个有意思的反差。同一批人在三个小时里展示了全世界最大规模的神经网络训练体系,然后告诉你绝大多数问题不该用神经网络解决。

仿真器的三层结构

关于仿真用了什么技术,回答很坦诚。最底层就是传统游戏技术,实时光栅化,和 GTA 那类东西相似。往上是实时光线追踪,这部分的成果是昨晚才跑出来的,演示画面底下那个小星号就是指这个。再往上是神经渲染,还在探索阶段。

他们的目标说得很明确:让模型永远无法在仿真器上过拟合。神经渲染是最后那颗樱桃,把真实度推到网络分辨不出来的程度。

目前做的接近那篇 photorealism enhancement 论文的思路,但他们能做的比论文多得多。标注数据更多、算力更多、对环境的控制更强,还有一批人专门保证它能实时运行。理想状态是完全用仿真训练一切,但马斯克说这永远不会成为必要,因为真实世界的数据量没人比他们多。

仿真存在的意义是填缝。真实世界里出现过的怪事,挑几个样本,造一千个变体,把那一类空白补上。

有提问者追问仿真到真实的域适应。仿真负责人说,目前的仿真视频是直接拿去训练的,但只占数据集里很小的一块,而且是有针对性的片段,评估永远只用真实视频。他们已经把车上同一套网络重训成“仿真 vs 真实”的判别器,用来指导美术决策。到目前为止没有出现过拟合仿真的问题,但随着数据规模上去,迟早会有,那时候就得靠神经渲染把这个边界推得更远。

马斯克把整套逻辑收在一个概念上:九的进军。从 99% 安全到 99.9%,再到 99.999999%,每多一个九,难度提升一个数量级。而且这里有个残酷的自反性:车越擅长避免事故,事故样本就越少,训练集就越小。所以他们不得不在仿真里制造事故,设定场景,5 米距离,20 米每秒,做什么动作能让伤害概率最小。还得模拟逆行车辆,那种情况偶尔发生,但不够频繁。

“计算机快,人类慢”

有人问自动驾驶怎么在乘员安全和行人安全之间做取舍。

马斯克的回答绕开了伦理框架,直接讲物理:从计算机的角度看,世界上所有东西都移动得很慢。人觉得快,机器不觉得。所以这个两难在现实中出现的概率很低。会发生,但极其罕见。

他给了一个对照。60 帧每秒、完全光追、还叠加神经网络增强的画面,人手工渲染一帧要一个月,还未必没有错。计算机一秒做 60 帧。火箭也是同一个道理,他们当年接了个摇杆想让人手动操控入轨,需要 6 到 7 赫兹的反应频率,没人做得到。

关于硬件。马斯克说 FSD 计算机(HW3)足以实现远超人类的安全水平,具体数字他给了 200% 到 300% 更好。下一代硬件大概会随 Cybertruck 推出,也许一年左右,能力大约 4 倍。硬件代际更替带来的提升是有上限的,更难的题目在另一头:能不能把安全水平从 300% 提到 1000%。他还补了一句让人踏实的话:路上开车的什么人都有,你不需要是全世界最好的司机才能上路。

关于对抗性攻击,比如有人举着印了停车标志的 T 恤去闪车。马斯克说自己就有这么一件 T 恤,试过,车会停下来。这类攻击一旦被发现,就把对应样本拿去训练,让网络学会判断“这是穿着 T 恤的人,衣服上有停车标志,大概不是真的标志牌”。

不确定性与影子模式

规划怎么处理不确定性,Ashok 拆成了几层。

首先是多模态未来。不是给周围每辆车挑一个最可能的意图,而是同时保留多种可能,用物理和运动学判断对方在做出干扰我们的动作之前还有多少余量,然后才行动。有疑问就保守让行。但保守有上限,太保守的车没人愿意坐,某个点上必须果断,而果断之前要确认对方有能力让行、并且行为合理。

第二层是影子模式。任何新东西进车队之前,先在后台跑一遍,看这套网络在这个场景里会怎么做。驾驶员实际上一直在训练网络。如果网络会往右打方向,而人往左打了,这个差异就是一条宝贵的数据。当然前提是人没撞车,撞了就不算。

第三层是不确定性估计,比如 flickr 这类指标。一旦判断自己看不清,就减速,先获取信息再行动。用马斯克的话说,理想状态是车开多快与它有多确信成正比,知道得越少开得越慢。

有人问,如果只能改进一个组件,哪个收益最大。马斯克说这是个曲线下面积的问题,改任何一处都会抬高整条曲线。短期最明确的是让所有网络都用上环视视频,现在还有一些遗留网络是单帧的,正在快速改。但他强调,比技术更缺的东西在别处:

很多拼图已经齐了,缺的是更多强的人来把这件事做成。这也是我们办这场活动的真正原因。(马斯克)

关于遗忘、对齐和那个电梯比喻

关于灾难性遗忘和在线持续学习。Ashok 说他们没怎么碰到文献里那些问题,因为压根没做太多持续学习。流程是训练、在若干场景上验证、广泛评估、确认稳定,然后推送到车上。要进车队的东西必须是可被大规模评估的稳定物。

但个性化会有。马斯克举的例子很生活化:就像你对 Uber 司机说的那些话,我赶飞机,麻烦开快点;或者不着急,慢慢来。驾驶风格会成为设置项。

关于 AI 对齐。马斯克没有回避:

我们应该对 AI 感到担忧。这里我们在做的是窄 AI,让车开得比人好,让人形机器人能做基础的事情。一旦开始触及超人类智能,那就不好说了。(马斯克)

他给特斯拉定的目标是做出有用的、人们喜欢的、明确是好的 AI。

最后一段关于相机的提问,他顺势给出了一个物理直觉很到位的类比。有人问摄像头相比人眼太粗糙,有没有考虑事件相机。

马斯克说,HW4 会有下一代相机,但当前这代相机远没有到极限,用现在的相机和算力实现远超人类的安全,他有信心。然后是那个比喻:

以前的电梯是有操作员的,坐在那儿扳一个大继电器开关。人会累,会走神,偶尔犯错,把人切成两半。后来电梯自动化了,你走进一栋一百层的大楼,按下按钮,什么都不用想。

车会走同一条路。所有车最终都会是自动的。当然还会有人开汽油车、开手动挡,就像今天还有人骑马。

我相信未来的人回头看会说,天啊,我们居然得自己开车,这太不可思议了。(马斯克)

如果你只有半小时,该从这里切入

这场发布会的信息密度不均匀,三个多小时里真正有迁移价值的部分集中在几个判断上。给不同角色几条具体的路径:

  • 做感知或多模态的工程师,直接看 Andrej 讲 Transformer 从图像空间转向量空间那一段。多摄像头融合、位置编码初始化 query、整流虚拟相机层,这三件事的组合是整场最扎实的工程细节,比任何综述都具体。
  • 做规划控制的,重点看 Ashok 用 A* 对比 MCTS 的那个停车问题。40 万节点降到 288 个节点,这个对比把“神经网络提供价值函数”从口号变成了可验证的数字。
  • 做数据基础设施的,把自动标注那一段反复看。利用离线能看见未来这一点做轨迹关联、做精确 kinematics,成本几乎为零,收益极大。这条工程直觉很容易被漏掉。
  • 评估仿真方案是否值得投入的,记住 Ashok 给的三条判据:数据难获取、数据难标注、行为闭环依赖车辆动作。三条都不满足,就不必上仿真。
  • 做训练基础设施或芯片的,看 Ganesh 讲节点尺寸怎么定的那一段。从延迟和带宽反推节点大小,再往上逐层放大,这个推导顺序比任何参数表都有价值。
  • 任何打算“用机器学习解决问题”的人,先看马斯克那段劝退。他的标准是:除非你不得不用,否则别用。把这句话贴在显示器上,能省掉大量返工。
  • 判断这件事的时间表,看九的进军这个概念。每多一个九,难度涨一个数量级,而越往后真实数据越稀缺。这意味着剩下的时间不是线性的,任何人给出的具体日期都值得打折。

最后一个建议更像提醒。这场发布会里技术含量最高的部分,在于他们反复承认难、承认没做完、承认要解决的问题还很多。Andrej 说架构还在演进,Ganesh 说编译器问题还没解决,马斯克说 99.9% 的情况下你不需要机器学习。

一个组织如果能在对外场合持续说这种话,它大概知道自己在做什么。

YouTube 平台字幕(自动或人工),可能有识别错误,人名与数字请以视频为准。已按语句合并成段落,并去掉 um、uh 等口头填充词。 文稿来源:YouTube(自动/人工字幕)。

0:02[Music] [Applause] [Music] [Applause] [Music] so [Music] i'm not the man you want you've made a mistake i'm in touch with technological advances pretty well everywhere and i can tell you that no one has produced anything like this [Music]

16:13so [Music] oh [Music] so [Music] wow [Music] armed do [Music]

19:48i'm armed [Music] so [Music] so [Music] [Applause] [Music] so [Music]

29:17so [Music] [Applause] [Music] [Applause] [Music] so [Music]

31:50so [Music] so [Music] so [Music] so [Music] i'm not the man you want you've made a i'm in touch with mistake advisors pretty well everywhere and i can tell you that no one has produced anything like this [Music]

39:21so [Music] so [Music] so [Music] [Applause] [Music] [Applause] [Music] [Applause] [Music]

41:31foreign so [Music] [Music] so [Music] you [Music] armed [Music]

45:31[Music] armed [Music] do [Music] hello everyone sorry for the delay thanks for coming and sorry i had some technical difficulties really neat ai for this so what we want to show today is that tesla is much more than an electric car company that we have deep ai activity in hardware on the inference level on the training level and basically we i think we're i think arguably the leaders in real world ai as it applies to the real world and those of you who have seen the full self-driving beta i can appreciate the rate at which the tesla neural net is learning to to drive and so this is a particular application of ai but i think there's there's more

48:01there are more applications down the road that will make sense and we'll talk about that later in the presentation but yeah we basically want to encourage anyone who is interested in solving real-world ai problems at either the hardware or the software level to join tesla or consider drawing tesla so let's see we'll start off with andre

48:30great okay hi everyone welcome my name is andre and i am i lead the vision team here at tesla autopilot and i'm incredibly excited to be here to kick off this section giving you a technical deep dive into the autopilot stack and showing you all the under the hood components that go into making the car drive all by itself so we're going to start off with the vision component here now in division component what we're trying to do is we're trying to design a neural network that processes the raw information which in our case is the eight cameras that are positioned around the vehicle and they send us images and we need to process that in real time into what we

49:21call the vector space and this is a three-dimensional representation of everything you need for driving so this is the three-dimensional positions of lines edges curbs traffic signs traffic lights cars their positions orientations depth velocities and so on so here i'm showing a video of actually hold on apologies so here i'm showing the video of the raw inputs that come into the stack and then neural processes that into the vector space and you are seeing parts of that vector space rendered in the instrument cluster on the car

50:10now what i find kind of fascinating about this is that we are effectively building a synthetic animal from the ground up so the car can be thought of as an animal it moves around it senses the environment and you know acts autonomously and intelligently and we are building all the components from scratch in-house so we are building of course all of the mechanical components of the body the nervous system which is all the electrical components and for our purposes the brain of the autopilot and specifically for this section the synthetic visual cortex now the biological visual cortex actually has quite intricate structure and a number of areas

50:43that organize the information flow of this brain and so in particular in our in your visual cortices the information hits the light hits the retina it goes through the lgn all the way to the back of your visual cortex goes through areas v1 v2 v4 the it the venture on the dorsal streams and the information is organized in a certain layout and so when we are designing the visual cortex of the car we also want to design the neural network architecture of how the information flows in the system so the processing starts in the beginning when light hits our artificial retina and we are going to process this information with neural networks now i'm

51:21going to roughly organize this section chronologically so starting off with some of the neural networks and what they looked like roughly four years ago when i joined the team and how they have developed over time so roughly four years ago the car was mostly driving in a single lane going forward on the highway and so it had to keep lane and it had to keep distance away from the car in front of us and at that time all of processing was only on individual image level so a single image has to be analyzed by a neural net and make little pieces of the vector space process that into little pieces of the vector space so this processing took the following shape we take a 1280

51:57by 960 input and this is 12 bit integers streaming in at roughly 36 hertz now we're going to process that with the neural network so instantiate a feature extractor backbone in this case we use residual neural networks so we have a stem and a number of residual blocks connected in series now the specific class of resnets that we use are regnets because we like this is like a very rec nets offer a very nice design space for neural networks because they allow you to very nicely trade off latency and accuracy now these reg nets give us as an output a number of features at different resolutions in different scales so in particular on the very bottom of this

52:34feature hierarchy we have very high resolution information with very low channel counts and all the way at the top we have low spatial low resolution especially but high channel counts so on the bottom we have a lot of neurons that are really scrutinizing the detail of the image and on the top we have neurons that can see most of the image and a lot of that context have a lot of that scene context we then like to process this with feature pyramid networks in our case we like to use by fbns and they get the mult they get to multiple scales to talk to each other effectively and share a lot of information so for example if you're a neuron all the way down in the

53:07network and you're looking at a small patch and you're not sure this is a car or not it definitely helps to know from the top players that hey you are actually in the vanishing point of this highway and so that helps you disambiguate that this is probably a car after a buy fbn and a feature fusion across scales we then go into task specific heads so for example if you are doing object detection we have a one stage yolo like object detector here where we initialize a raster and there's a binary bit per position telling you whether or not there's a car there and then in addition to that if there is here's a bunch of other attributes you might be interested in so the x y with

53:41height offset or any of the other attributes like what type of a car is this and so on so this is for the detection by itself now very quickly we discovered that we don't just want to detect cars we want to do a large number of tasks so for example we want to do traffic light recognition and detection a lane prediction and so on so very quickly we conversion this kind of architectural layout where there's a common shared backbone and then branches off into a number of heads so we call these therefore hydronets and these are the heads of the hydra now this architectural layout has a number of benefits so number one because of the feature sharing we can

54:18amortize the forward pass inference in the car at test time and so this is very efficient to run because if we had to have a backbone for every single task that would be a lot of backbones in the car number two this decouples all of the tasks so we can individually work on every one task in isolation and for example we can we can upgrade any of the data sets or change some of the architecture of the head and so on and you are not impacting any of the other tasks and so we don't have to revalidate all the other tasks which can be expensive and number three because there's this bottleneck here and features what we do fairly often is that we actually cache these features to disk

54:52and when we are doing these fine tuning workflows we only fine-tune from from the cached features up and only find you in the heads so most often in terms of our training workflows we will do an end-to-end training run once in a while where we train everything jointly then we cache the features at the multiscale feature level and then we fine-tune off of that for a while and then end-to-end train once again and so on so here's the kinds of predictions that we were obtaining i would say several years ago now from one of these hydronitis so again we are processing individual images there we go we are processing just individual image and we're making a

55:30large number of predictions about these images so for example here you can see predictions of the stop signs the stop lines the lines the edges the cars the traffic lights the curbs here whether or not the car is sparked all of the static objects like trash cans cones and so on and everything here is coming out of the net here in this case out of the hydra net so that was all fine and great but as we worked towards fsd we quickly found that this is not enough so where this first started to break was when we started to work on smart summon here i am showing some of the predictions of only the curb detection task and i'm showing it now for every one of the cameras

56:07so we'd like to wind our way around the parking lot to find the person who is summoning the car now the problem is that you can't just directly drive on image space predictions you actually need to cast them out and form some kind of a vector space around you so we attempted to do this using c plus and developed what we call the occupancy tracker at the time so here we see that the curb detections from the images are being stitched up across camera scenes camera boundaries and over time now there are two pro two major problems i would say with the setup number one we very quickly discovered that tuning the occupancy tracker and all of its hyper

56:41parameters was extremely complicated you don't want to do this explicitly by hand in c plus you want this to be inside the neural network and train that end to end number two we very quickly discovered that the image space is not the correct output space you don't want to make predictions in image space you really want to make it directly in the vector space so here's a way of illustrating the issue so here i'm showing on the first row the predictions of our curves and our lines in red and blue and they look great in the image but once you cast them out into the vector space things start to look really terrible and we are not going to be able to drive on

57:15this so you see how the predictions are quite bad in vector space and the reason for this fundamentally is because you need to have an extremely accurate depth per pixel in order to actually do this projection and so you can imagine just how high of the bar it is to predict that depth so accurately in these tiny in every single pixel of the image and also if there's any occluded area where you'd like to make predictions you will not be able to because it's not an image space concept in that case so we very quickly real of the other problems with this by the way is also for object detection if you are only making predictions per camera then sometimes you will encounter cases like

57:54this where a single car actually spans five of the eight cameras and so if you are making individual predictions then no single camera since sees all of the car and so obviously you're not going to be able to do a very good job of predicting that whole car and it's going to be incredibly difficult to fuse these measurements so we have this intuition that what we'd like to do instead is we'd like to take all of the images and simultaneously feed them into a single neural net and directly output in vector space now this is very easily said much more difficult to actually achieve but roughly we want to lay out a neural net in this way where we process every single image with

58:28a backbone and then we want to somehow fuse them and we want to re-represent it the the features from image space features to directly some kind of vector space features and then go into the decoding of the head now so there are two problems with this problem number one how do you actually create the neural network components that do this transformation and you have to make it differentiable so that end-to-end training is possible and number two the if you want vector space predictions from your neural net you need vector specif based data sets so just labeling images and so on is not going to get you there you need vector space labels we're

59:05going to talk a lot more about problem number two later in the talk for now i want to focus on the neural network architectures so i'm going to deep dive into problem number one so here's the rough problem right we're trying to have this bird's eye view prediction instead of image space predictions so for example let's focus on a single pixel in the output space in yellow and this pixel is trying to decide am i part of a curb or not as an example and now where should the support for this kind of a prediction come from in the image space well we know roughly how the cameras are positioned and their extrinsics and intrinsics so we can roughly project

59:38this point into the camera images and you know the evidence for whether or not this is a curve may come from somewhere here in the images the problem is that this projection is really hard to actually get correct because it is a function of the road surface the road surface could be sloping up or sloping down or also there could be other data dependent issues for example there could be occlusion due to a car so if there's a car occluding this this viewport this this part of the image then actually you may want to pay attention to a different part of the image not the part where it projects and so because this is data dependent it's really hard to have a

1:00:08fixed transformation for this component so in order to solve this issue we use a transformer to represent this space and this transformer uses multi-headed self-attention and blocks off it in this case actually we can get away with even a single block doing a lot of this work and effectively what this does is you initialize a raster of the size of the output space that you would like and you tile it with positional encodings with size and coses in the output space and then these get encoded with an mlp into a set of query vectors and then all of the images and their features also emit their own keys and values and then the queries keys and values feed into the multi-headed

1:00:47self-attention and so effectively what's happening is that every single image piece is broadcasting in its key what it is what is it a part of so hey i'm part of a pillar in roughly this location and i'm seeing this kind of stuff and that's in the key and then every query is something along the lines of hey i'm a pixel in the output space at this position and i'm looking for features of this type then the keys and the queries interact multiplicatively and then the values get pulled accordingly and so this re-represents the space and we find this to be very effective for this transformation so if you do all of the engineering correctly this again is

1:01:20very easily said difficult to do you do all of the engineering correctly there's one more there's no problem actually before i'm not sure what's up with the slides so one more thing you have to be careful with some of the details here when you are trying to get this to work so in particular all of our cars are slightly cockeyed in a slightly different way and so if you're doing this transformation from image space to the output space you really need to know what your camera calibration is and you need to feed that in somehow into the neural net and so you could definitely just like concatenate the camera calibrations of all of the images and

1:01:54somehow feed them in with an mlp but actually we found that we can do much better by transforming all of the images into a synthetic virtual camera using a special rectification transform so this is what that would look like we insert a new layer right above the image rectification layer it's a function of camera calibration and it translates all of the images into a virtual common camera so if you were to average up a lot of repeater images for example which faced at the back you would without doing this you would get a kind of a blur but after doing the rectification transformation you see that the back mirror gets really crisp so once you do this this improves the

1:02:32performance quite a bit so here are some of the results so on the left we are seeing what we had before and on the right we're now seeing significantly improved predictions coming directly out of the neural net this is a multi-camera network predicting directly in vector space and you can see that it's basically night and day you can actually drive on this and this took some time and some engineering and incredible work from the ai team to actually get this to work and deploy and make it efficient in the car this also improved a lot of our object detection so for example here in this video i'm showing single camera predictions in orange and multi-camera

1:03:08predictions in blue and basically if you if you can't predict these cars if you are only seeing a tiny sliver of a car so your detections are not going to be very good and their positions are not going to be good but a multi-camera network does not have an issue here's another video from a more nominal sort of situation and we see that as these cars in this tight space cross camera boundaries there's a lot of jank that enters into the predictions and basically the whole setup just doesn't make sense especially for very large vehicles like this one and we can see that the multi-camera networks struggle significantly less with these kinds of predictions

1:03:40okay so at this point we have multi-camera networks and they're giving predictions directly in vector space but we are still operating at every single instant in time completely independently so very quickly we discovered that there's a large number of predictions we want to make that actually require the video context and we need to somehow figure out how to feed this into the net so in particular is this car parked or not is it moving how fast is it moving is it still there even though it's temporarily occluded or for example if i'm trying to predict the road geometry ahead it's very helpful to know of the signs or the road markings that i saw 50

1:04:11meters ago so we try to develop we try to insert video modules into our neural network architecture and this is kind of one of the solutions that we've converged on so we have the multi-scale features as we had them from before and what we are going to now insert is a feature queue module that is going to cache some of these features over time and then a video module that is going to fuse this information temporally and then we're going to continue into the heads that do the decoding now i'm going to go into both of these blocks one by one also in addition notice here that we are also feeding in the kinematics this is basically the velocity and the acceleration that's

1:04:45telling us about how the car is moving so not only are not only are we going to keep track of what we're seeing from all the cameras but also how the car has traveled so here's the feature queue and the rough layout of it we are basically concatenating these features over time and the kinematics of how the car has moved and the positional encodings and that's being concatenated encoded and stored in a feature queue and that's going to be consumed by a video module now there's a few details here again to get right so in particular with respect to the pop and push mechanisms and when do you push and how and especially when do you push basically so

1:05:20here's a cartoon diagram illustrating some of the challenges here there's going to be the ego cars coming from the bottom and coming up to this intersection here and then traffic is going to start crossing in front of us and it's going to temporarily start occluding some of the cars ahead and then we're going to be stuck at this intersection for a while and just waiting our turn this is something that happens all the time and is a cartoon representation of some of the challenges here so number one with respect to the feature queue and when we want to push into a queue obviously we'd like to have some kind of a time-based queue where for example we enter the features

1:05:51into the queue say every 27 milliseconds and so if a car gets temporarily occluded then the neural network now has the power to be able to look and reference the memory in time and and learn the association that hey even though this thing looks occluded right now there's a record of it in my previous features and i can use this to still make a detection so that's kind of like the more obvious one but the one that we also discovered is necessary in our case is for example suppose you're trying to make predictions about the road surface and the road geometry ahead and you're trying to predict that i'm in a turning lane and the lane next to us is going

1:06:25straight then it's really necessary to know about the line markings and the signs and sometimes they occur a long time ago and so if you only have a time-based queue you may forget the features while you're waiting at your red light so in addition to a time-based q we also have a space-based view so we push every time the car travels a certain fixed distance so some of these details actually can matter quite a bit and so in this case we have a time based key and a space base key to cache our features and that continues into the video module now for the video module we looked at a number of possibilities of how to fuse this information temporally so we looked

1:07:00at three-dimensional convolutions transformers axial transformers in an effort to try to make them more efficient recurrent neural networks over a large number of flavors but the one that we actually like quite a bit as well and i want to spend some time on is a spatial recurrent neural network video module and so what we're doing here is because of the structure of the problem we're driving on two-dimensional surfaces we can actually organize the hidden state into a two-dimensional lattice and then as the car is driving around we update only the parts that are near the car and where the car has visibility so as the car is driving around we are using the kinematics to

1:07:33integrate the position of the car in the hidden features grid and we are only updating the rnn at the points where where we have that are nearby us sort of so here's an example of what that looks like here what i'm going to show you is the car driving around and we're looking at the hidden state of this rnn and these are different channels in the hidden state so you can see that this is after optimization and training this neural net you can see that some of the channels are keeping track of different aspects of the road like for example the centers of the road the edges the lines the road surface and so on here's another cool video of this so

1:08:16this is looking at the mean of the first 10 channels in the hidden state for different traversals of different intersections and all i want you to see basically is that there's cool activity as the recurrent neural network is keeping track of what's happening at any point in time and you can imagine that we've now given the power to the neural network to actually selectively read and write to this memory so for example if there's a car right next to us and is occluding some parts of the road then now the network has a has the ability to not write to those locations but when the car goes away and we have a really good view then the recurring neural net

1:08:46can say okay we have very clear visibility we definitely want to write information about what's in that part of space here's a few predictions that show what this looks like so here we are making predictions about the road boundaries in red intersection areas in blue road centers and so on so we're only showing a few of the predictions here just to keep the visualization clean

1:09:10and yeah this is this is done by the spatial rnn and this is only showing a single clip a single traversal but you can imagine there could be multiple trips through here and basically number of cars a number of clips could be collaborating to build this map basically and effectively an hd map except it's not in a space of explicit items it's in a space of features of a recurrent neural network which is kind of cool i haven't seen that before the video networks also improved our object detection quite a bit so in this example i want to show you a case where there are two cars over there and one car is going to drive by and occlude them briefly so look at

1:09:47what's happening with the single frame and the video predictions as the cars pass in front of us yeah so that makes a lot of sense so a quick playthrough through what's happening when both of them are in view the predictions are roughly equivalent and you are seeing multiple orange boxes because they're coming from different cameras when they are occluded the single frame networks drop the detection but the video module remembers it and we can persist the cars and then when they are only partially occluded the single frame network is forced to make its best guess about what it's seeing and it's forced to make a prediction and it makes a really terrible prediction but the video

1:10:26module knows that there's only a partial that you know it has the information and knows that this is not a very easily visible part right now and doesn't actually take that into account we also saw significant improvements in our ability to estimate depth and of course especially velocity so here i'm showing a clip from our remove the radar push where we are seeing the radar depth and velocity in green and we were trying to match or even surpass of course the signal just from video networks alone and what you're seeing here is in orange we are seeing a single frame performance and in blue we are seeing again video modules and so you see that the quality

1:11:03of depth is much higher and for velocity the orange signal of course you can't get velocity out of a single frame network so we use we just differentiate depth to get that but the video module actually is basically right on top of the radar signal and so we found that this worked extremely well for us so here's putting everything together this is what our architectural roughly looks like today so we have raw images feeding on the bottom they go through rectification layer to correct for camera calibration and put everything into a common virtual camera we pass them through regnets residual networks to process them into a number of features at

1:11:39different scales we fuse the multi-scale information with by fbn this goes through transformer module to re-represent it into the vector space in the output space this feeds into a feature queue in time or space that gets processed by a video module like the spatial rnn and then continues into the branching structure of the hydra net with trunks and heads for all the different tasks and so that's the architecture roughly what it looks like today and on the right you are seeing some of its predictions sort of visualize both in a top-down vector space and also in images so definitely this architecture has definitely complexified from just a very simple

1:12:12image based single network about three or four years ago and continues to evolve it's definitely quite impressive now there's still opportunities for improvements that the team is actively working on for example you'll notice that our fusion of time and space is fairly late in neural network terms so maybe we can actually do earlier fusion of space or time and do for example cost volumes or optical flow-like networks on the bottom or for example our outputs are dense rasters and it's actually pretty expensive to post-process some of these dense rasters in the car and of course we are under very strict latency requirements so this is not ideal we

1:12:43actually are looking into all kinds of ways of predicting just the sparse structure of the road maybe like you know point by point or in some other fashion that is that doesn't require expensive post-processing but this basically is how you achieve a very nice vector space and now i believe ashok is going to talk about how we can run playing control on top of it [Applause]

1:13:11thank you andre hi everyone my name is ashok i lead the planning and controls auto labeling and simulation teams so like on dimension the visual networks take dense video data and then compress it down into a 3d vector space the role of the planner now is to consume this vector space and get the car to the destination while maximizing the safety comfort and the efficiency of the car even back in 2019 our planet was pretty capable driver it was able to stay in the lanes make lane changes as necessary and take exits off the highway but cdc driving is much more complicated radley there are structured lane lines vehicles do much more free from driving then the car

1:13:51has to respond to all of curtains and crossing vehicles and pedestrians doing funny things what is the key problem in planning number one the action space is very non-convex and number two it is high dimensional what i mean by non-convex is there can be multiple possible solutions that can be independently good but getting a globally consistent solution is pretty tricky so there can be pockets of local minima that the planning can get stuck into and secondly the high dimensionality comes because the car needs to plan for the next 10 to 15 seconds and needs to produce the position velocities and acceleration or this entire window this is a lot of parameters to produce

1:14:37at runtime discrete search methods are really great at solving non-context problems because they are discrete they can they don't get stuck in local minima whereas continuous function optimization can easily get stuck in local minima and produce poor solutions that are not great on the other end for high dimensional problems discrete search sucks because of the it is discrete it does not use any graded information so literally has to go and explore each point to know how good it is whereas continuous optimization use gradient-based methods to very quickly go to a good solution

1:15:10our solution to this printer problem is to break it down hierarchically first use a code search method to crunch down the non-convexity and come up with a convex corridor and then use continuous optimization techniques to make the final smooth trajectory let's see an example of how the search operates so here we're trying to do a lane change in this case the car needs to do two back to back lane changes to make the left turn up ahead for this the car searches over different maneuvers so the the first one it searches is a lane change that's close by but the car breaks pretty harshly so it's pretty uncomfortable the next maneuver it tries does the lane

1:15:58change bit late so it speeds up goes beyond the other car goes in front of the other cars and find it at the lane change but now it risks missing the left turn we do thousands of such searches in a very short time span because these are all physics-based models these features are very easy to simulate and in the end we have a set of candidates and we finally choose one based on the optimality conditions of safety comfort and easily making the turn so now the car has chosen this path and you can see that as the car executes this trajectory it pretty much matches what we had planned the cyan plot on the right side here and that one is the actual velocity of the car and the white

1:16:36line be underneath it is was the plan so we are able to plan for 10 seconds here and able to match that when you see in hindsight so this is a well-made plan when driving alongside other agents it's important to not just plan for ourselves but instead we have to plan for everyone jointly and optimize for the overall scenes traffic flow in order to do this what we do is we literally run the autopilot planner on every single relevant object in the scene here's an example of why that's necessary this is an auto corridor i'll let you watch the video for a second

1:17:25yeah that was autopilot driving an auto corridor going around parked cars cones and poles here there's a 3d view of the same thing the oncoming car arrives now and autopilot slows down a little bit but then realizes that we cannot yield to them because we don't have any space to our side but the other car can heal to us instead so instead of just blindly breaking here all of our reasons about that car has low enough velocity that they can pull over and should yield to us because we cannot yield to them and assertively makes progress a second oncoming car arrives now this vehicle has higher velocity and like i said earlier we literally run the autopilot planner for the other

1:18:02object so in this case we run the printer for them that object's plan now goes around their part their site's parked cars and then after they pass the parkours goes back to the right side of the road for them since we don't know what's in the mind of the driver we actually have multiple possible features for this car here one feature is shown in red the other one is shown as green the green one is a plan that yields to us but since this object's velocity and acceleration are pretty high we don't think that this person is going to yield to us and they are actually going to go around these sparked cars so autopilot decides that okay i have space here this person is definitely

1:18:35going to come so i'm going to pull over so as autopilot is pulling over we notice that that car has chosen to yield to us based on their yaw rate and their acceleration and autopilot immediately changes his mind and continues to make progress this is why we need to plan for everyone because otherwise we wouldn't know that this person is going to go around the other park cars and come back to their site if you didn't do this autopilot would be too timid and it would not be a practical self-driving car so now we saw how the search and planning for other people set up convicts valley finally we do a continuous optimization to produce the final trajectory that the planet

1:19:12needs to take here the grave thing is the convex corridor and we initialize the spline in heading and acceleration parameterized over the arc length of the plan and you can see that the countermeasuration continuously makes fine-grained changes to reduce all of its cost some of the costs for example are distance from obstacles traversal time and comfort for comfort you can see that the lateral acceleration plots on the right have nice trapezoidal shapes it's going to come up yeah here on the right side the green plot that's a nice trapezoidal shape and if you record on a human trajectory this is pretty much how it looked like the lateral jerk is also

1:19:47minimized so in summary we do a search for both us and everyone else in the scene we set up a context corridor and then optimize for a smooth path together these can do some really neat things like shown above but driving looks a bit different in other places like where i grew up from it's very much more unstructured cars and pedestrians cutting each other arch braking honking it's a crazy world we can try to scale up these methods but it's going to be really difficult to efficiently solve this at runtime what we instead want to do is use learning based methods to efficiently solve them and i want to show why this is true so we're going to go from this

1:20:29complicated problem to a much simpler toy parking problem but still illustrates the core of the issue here this is a parking lot the ego car is in blue and needs to park in the green parking spot here so it needs to go around the curbs the park cars and the cones shown in orange here let's do this simple bass line it's a star the standard algorithm that uses a ladder space search and and the heuristic here is a distance the euclidean distance to the goal so you can see that it directly shoots towards the goal but very quickly gets trapped in a local minima and it backtracks from there and then searches a different path to try to go around this parkour

1:21:07eventually it makes progress and gets to the goal but ends up using 400 000 nodes for making this obviously this is a terrible heuristic we want to do better than this so if you added a navigation route to it and has the car to follow the navigation route while being close to the goal this is what happens the navigation route helps immediately but still when it enter encounters cones or other obstacles it basically that's the same thing as before backtracks and then searches that whole new path and the support search has no idea that these obstacles exist it literally has to go there check if it's in collision and if it's in collision back up

1:21:46the navigation heuristic helped but still this took 22 000 notes we can design more and more of these heuristics to help the search make go faster and faster but it's really tedious and hard to design a globally optimal heuristic even if you had a distance function from the cones that guided the search this would not this would not be this would only be effective for the single cone but what we need is a global global value function so instead of what we want to use is neural networks to give this heuristic for us the visual networks produces vector space and we have cars moving around in it this basically looks like a atari game and it's a multiplayer version so we

1:22:24can use techniques such as mu0 alpha zero et cetera that was used to solve go and other authority games to solve the same problem so we're working on neural networks that can produce state and action distributions that can then be plugged into multicolored research with various cost functions some of the cost functions can be explicit cost functions like distance like collisions comfort traversal time etc but they can also be interventions from the actual manual driving events we train such a network for this simple parking problem so here again same problem let's see how mcts researched us

1:23:01so here you notice that the planet is basically able to in one shot make progress towards the goal to notice that this is not even using a navigation heuristic just given the scene the planet is able to go directly towards the goal all the other options you're seeing are possible options it's not choosing any of them just using the option that directly takes it towards the goal the reason is that the neural network is able to absorb the global context of the scene and then produce a value function that effectively guides it towards the global minima as opposed to getting certain any local minima so this only takes 288 notes and several orders of magnitude less than what was

1:23:36done in the a star with the equilibrium distance heuristic so this is what a final architecture is going to look like the vision system is going to crush down the dense video data into a vector space it's going to be consumed by both an explicit planner and a neural network planner in addition to this the network panel can also consume intermediate features of the network together this produces a trajectory distribution and it can be optimized end to end both with explicit cost functions and human intervention and other imitation data this then goes into explicit planning function that does whatever is easy for that and produces the final steering and

1:24:10acceleration commands for the car with that we need to now explain how we train these networks and for training these networks we need large data sets and wait i'm ready to speak briefly about manual labeling yeah so the data the story of data sets is critical of course so far we've talked only about neural networks but neural networks only establish an upper bound on your performance many of these neural networks they have hundreds of millions of parameters and these hundreds of millions of parameters they have to be set correctly if you have a bad setting of parameters it's not going to work so neural networks are just an upper bound you also need massive data sets to actually

1:24:56train the correct algorithms inside them now in particular i mentioned we want data sets directly in the vector space and so the really the question becomes how can you accumulate because our networks have hundreds of millions of parameters how do you accumulate millions and millions of vector space examples that are clean and diverse to actually train these neural networks effectively now so there's a story of data sets and how they've evolved on the side of all of the models and developments that we've achieved now in particular when i joined roughly four years ago we were working with a third party to obtain a lot of our data sets now unfortunately we found very

1:25:31quickly that working with a third party to get data sets for something this critical was just not going to cut it the latency of working with a third party was extremely high and honestly the quality was not amazing and so in the spirit of full vertical integration at tesla we brought all of the labeling in-house and so over time we've grown more than one thousand person data labeling org that is full of professional labelers who are working very closely with the engineers so actually they're here in the u.s and co-located with the engineers here in the area as well and so we work very closely with them and we also build all of the infrastructure for them from

1:26:06scratch ourselves so we have a team we are going to meet later today that develops and maintains all of this infrastructure for data labeling and so here for example i'm showing some of the screenshots of some of the latency throughput and quality statistics that we maintain about all of the labeling workflows and the individual people involved and all the tasks and how the numbers of labels are growing over time

1:26:28so we found this to be quite critical and we're very proud of this now in the beginning roughly three or four years ago most of our labeling was in image space and so you can imagine that this is taking quite some time to annotate an image like this and this is what it looked like where we are sort of drawing polygons and polylines on top of on top of these single individual images as i mentioned we need millions of vector space labels so this is not going to cut it so very quickly we graduated to three-dimensional or four-dimensional labeling where we are directly labeling in vector space not in individual images so here what i'm showing is a clip and you are seeing a very

1:27:06small reconstruction you're about to see a lot more reconstructions soon but it's very small reconstruction of the ground plane on which the car drove and a little bit of the point cloud here that was reconstructed and what you're seeing here is that the labeler is changing the labels directly in vector space and then we are reprojecting those changes into camera images so we're labeling directly in vector space and this gave us a massive increase in throughput for a lot of our labels because you're labeled once in 3d and then you get to reproject but even this we realized was actually not going to cut it so because people and computers have

1:27:41different pros and cons so people are extremely good at things like semantics but computers are very good at geometry reconstruction triangulation tracking and so really for us it's much more becoming a story of how do humans and computers collaborate to actually create these vector space data sets and so we're gonna not talk about auto labeling which is some of the infrastructure we've developed for labeling these clips at scale [Applause]

1:28:08hi again so even though we have lots of human laborers the amount of training data needed for training with networks significantly outnumbers them so we try to invest in a massive auto labeling pipeline here's an example of how we label a single clip a clip is a entity that has dense sensor data like videos imu data gps odometry etc this can be 45 second to a minute long these can be uploaded by our own engineering cars or from customer cars we collect these clips and then send them to our servers where we run a lot of neural networks offline to produce intermediate results like segmentation masks depth point matching etc this then goes through a lot of robotics and a

1:28:49algorithm to produce a final set of labels that can be used to train the networks one of the first tasks we want to label is the road surface typically we can use splines or meshes to represent the road surface but those are because of the topology restrictions are not differentiable and not amenable to producing this so what we do instead is in the style of neural radiance fields work from last year which is quite popular so we use an implicit representation to represent the road surface here we are querying xy points on the ground and asking for the network to predict the height of the ground surface along with various semantics such as curves lane boundaries road surface

1:29:26rival space etc so given a single xy we get a z together these make a 3d point and they can be reprojected into all the camera views so we make millions of such queries and get lots of points these points are reprojected into all the camera views in we are showing on the top right here one such camera image with all these points reprojected now we can compare this point reprojected point with the image space prediction of the segmentations and jointly optimizing this or all the camera views was across space and time produced an excellent reconstruction here's an example of how that looks like so here this is an optimized road surface that reproduction to the eight

1:30:08cameras that the car has and across all of time and you can see how it's consistent across both space and time so a single car driving through some location can sweep out some patch around the trajectory using this technique but we don't have to stop there so here we collect collected different clips from the same location from different cars maybe and each of them sweeps out some part of the road cool thing is we can bring them all together into a single giant optimization so here these 16 different trips are organized using aligned using various features such as roadages lane lines all of them should agree with each other and also agree with all of their image space

1:30:51observations together this is this produces an effective way to label the road surface not just where the car drove but also in other locations that it hasn't driven it again the point of this is not to just build hd maps or anything like that it's only to label the clips through these intersections so we don't have to maintain them forever as long as the labels are consistent with the videos that they were collected at optionally then humans can come on top of this and clean up any noise or additional metadata to make it even richer we don't have to stop at just the road surface we can also arbitrarily reconstruct 3d static obstacles here this is a reconstructed 3d point cloud

1:31:30from our cameras the main innovation here is the density of the point cloud typically these points require texture to form associations from one frame to the next frame but here we are able to produce these points even on textual surfaces like the road surface or walls and this is really useful to annotate arbitrary obstacles that we can see on the scene in the world

1:31:55one more cool advantage of doing all of this on server on the servers offline is that we have the benefit of hindsight this is a super useful hack because say in the car then the network needs to produce the velocity it just has to use the historical information and guess what the velocity is but here we can look at both the history but also the future and basically cheat and get the correct answer of the kinematics like velocity acceleration etc one more advantage is that we can have different tracks but we can switch them together that even through occlusions because we know the future we have future tracks we can match them and then associate them so here you can see the

1:32:31pedestrians on the other side of the road are persisted even through multiple occlusions by these cars this is really important for the planner because the planner needs to know if it's if it saw someone it still needs to account for them even then they are occluded so this is a massive advantage

1:32:49combining everything together we can produce these amazing data sets that annotate all of the road texture or the static objects and all of the moving objects even through occlusions producing excellent kinematic labels all you can see how the cards turn smoothly produce really smooth labels or the pedestrians are consistently tracked the park cars obviously zero velocity so we can also know that they are parked so this is huge for us this is one more example of the same thing you can see how everything is consistent we want to produce a million such labeled clips and train our video multicam video networks with such large data set and really crush this problem

1:33:31we want to get the same view that's consistent that we're seeing here in the car we started our first exploration of this with the remove the data project we removed it in a very top a short time span i think within three months in the early days of the network we noticed for example in low security conditions the network can suffer understandably because obviously this truck just dumped a bunch of snow on us and it's really hard to see but we should still remember that this car was in front of us but our networks early on did not do this because of the lack of data in such conditions so what we did we added the fleet to produce lots of similar clips

1:34:07and the fleet responded it did so it produces is it play yeah it produces lots of video clips where shit's falling out of all other vehicles and we send this through our auto living pipeline that was able to label 10k clips in within a week this would have taken several months with humans labeling every single clip here so we did this for 200 of different conditions and we were able to very quickly create large data sets and that's how we were able to remove this so once we train the networks with this data you can see that it's totally working and keeps the memory that the subject was there and provides this so finally we wanted to actually get a

1:34:53cyber truck into a data set for the remote radar can you all guess where we got this clip from i'll give you a moment someone said it yes yes it's rendered it's our simulation it was hard for me to tell initially and i if i may if i may say so myself it looks pretty it looks very pretty

1:35:15so yeah in addition to auto labeling we also invest heavily in using simulation for labeling our data so this is the same scene as seen before but from a different camera angle so a few things that i wanted to point out for example the ground surface it's not a plane asphalt there are lots of cars and cracks and tower seams there's some patchwork done on top of it vehicles move realistically the truck is articulated even goes over the curb and makes a wide turn the other cars behave smartly they avoid collisions go around cars and also smooth and actual great smooth brake and accelerate smoothly the car here with the logo on the top is autopilot actually auto part is driving

1:35:59that car and it's making unproductive left hand and since it's a simulation it starts from the vector space so it has perfect labels here we show a few of the labels that we produce these are vehicle cuboids with kinematics depth surface normals segmentation but andre can name a new task that he wants next week and we can very quickly produce this because we already have a vector space and we can write the code to produce these labels very very quickly so when does simulation help it helps number one when the data is difficult to source as large as our fleet is it can still be hard to get some crazy scenes like this couple and their dog running on the highway while

1:36:39there are other high-speed cars around this is a pretty rare scene i'd say but still can happen and autopilot still needs to handle it when it happens when data is difficult to label there are hundreds of pedestrians crossing the road this could be a mountain downtown people crossing the road it's going to take several hours for humans to label this clip and even for automatic labeling algorithms this is really hard to get the association right and it can produce like bad velocities but in simulation this is trivial because you already have the objects you just have to spit out the cuboids and the velocities and also finally when we introduce

1:37:12closed loop behavior where the cars needs to be in a determining situation or the data depends on the actions this is pretty much the only way to get it reliably all this is great what's needed to make this happen number one accurate sensor simulation again the point of the simulation is not to just produce pretty pictures it needs to produce what the camera in the car would see and other sensors would see so here we are stepping through different exposure settings of the real camera on the left side and the simulation on the right side we're able to pretty much match what the real cameras do in order to do this we had to model a lot of the properties of the camera in

1:37:54our sensor simulation starting from sensor noise motion blur optical distortions even headlight transmissions even like diffraction patterns of the windshield etcetera we don't use this just for the autopilot software we also use it to make hardware decisions such as lens design camera design sensor placement even headlight transmission properties

1:38:19second we need to render the visuals in a realistic manner you cannot have what in the game industry called jaggies these are aliasing artifacts that are dead giveaway that this is simulation we don't want them so we go through a lot of paints to produce a nice special temple a special temporal anti-aliasing we also are working on neural rendering techniques to make this even more realistic

1:38:47yeah in addition we also use ray tracing to produce realistic lighting and global illumination okay that's the last of the cop cars i think we obviously cannot have really just four or five cars because the network will easily overfit because it knows the sizes so we need to have realistic assets like the moose on the road here we have thousands of assets in our library and they can wear different shirts and actually can move realistically so this is really cool we also have a lot of different locations mapped and created to create these sim environments we have actually 2000 miles of road built and this is almost the length of the roadway from the east coast the

1:39:25west coast of the united states which i think is pretty cool in addition we have built efficient tooling to build several miles more on a single day on a for a single artist but this is just tip of the iceberg actually most of the data that we use to train is created procedurally using algorithms as opposed to artists making these simulation scenarios so these are all procedurally created roads with lots of parameters such as curvature various varying trees cones poles cars with different velocities and the interaction produce an endless stream of data for the network but a lot of this data can be boring because the network might already get it

1:40:02correct so what we do is we use also ml based techniques to basically put up the network to see where it's failing at and create more data around the failure points of the network so this is in closed loop trying to make the network performance be better we don't want to stop there so actually we want to create recreate any failures that happens to the autopilot in simulation so that we can hold autopilot to the same bar from then on so here on the left side you are seeing a real clip that was collected from a car it then goes through our auto labeling pipeline to produce a 3d reconstruction of the scene along with all the moving objects with this combined with the original

1:40:42visual information we recreate the same scene synthetically and create a simulation scenario entirely out of it so and then when we replay autopilot on it autopilot can do entirely new things and we can form new worlds new outcomes from the original failure this is amazing because we really don't want autopilot to fail and when it fails we want to capture it and keep it to that bar

1:41:07not just that we can actually take the same approach that we said earlier and take it one step further we can use neural rendering techniques to make it look even more realistic so we take the original original video clip we create a synthetic simulation from it and then apply neural rendering techniques on top of it and it produces this which looks amazing in my opinion because this one is very realistic and looks almost like it was captured by the actual cameras they saw results from last night because it was cool and we wanted to present it but yeah yeah i'm very excited for what sim can achieve but this is not all because network's trained in the car already

1:41:42used simulation data we used 300 million images with almost half a billion labels and we want to crush down all the tasks that are going to come up for the next several months with that i invite milan to see to explain how we scale these operations and really build a label factory and spit out millions of labels [Applause]

1:42:09all right thanks ashok hey everyone i'm milan i'm responsible for the integration of our neural networks in the car and for most of our neural network training and evaluation infrastructure and so tonight i'd just like to start by giving you some perspective into the amount of compute that's needed to power this type of data generation factory and so in the specific context of the push we went through as a team here a few months ago to get rid of the dependency on the radar sensor for the pilot we generated over 10 billion labels across two and a half million clips and so to do that we had to scale our huge offline neural networks and our simulation engine across thousands of

1:42:48gpus and just a little bit shy of 20 000 cpu cars on top of that we also included over 2000 actual autopilot full self-driving computers in the loop with our simulation engine and that's our smallest compute cluster so i'd like to give you some idea of what it takes to take our neural networks and move them in the car and so the the two main constraints that we're working on there here are mostly latency and frame rate which are very important for safety but also to get proper estimates of acceleration and velocity of our surroundings and so the meat of the problem really is around the ai compiler that we write and extend here within the group that

1:43:36essentially maps the compute operations for my apart torch model to a set of dedicated and accelerated pieces of hardware and we do that while figuring out a schedule that's optimized for throat put while working on their severe sram constraints and so by the way we're not doing that just on one engine but on across two engines on the autopilot computer and the way we use those engines here at tesla is such that at any given time only one of them will actually output control commands to the vehicle while the other one is used as an extension of compute but those roles are interchangeable both at the hardware and software level so how do we interact quickly together

1:44:16as a group through these ai development cycles well first we have been scaling our capacity to evaluate our software neural network dramatically over the past few years and today we're running over a million evaluations per week on any code change that the team is producing and those evaluations runs on over three thousand actual footstar driving computers that are hooked up together in a dedicated cluster and so on top of this we've been developing really cool debugging tools and so here is a video of one of our tools which is helping developers iterate through the development of neural networks and comparing live the outputs from different revisions of a

1:44:56same neural network model as reiterating life through a video clips and so last but not least we've been scaling our neural network training compute dramatically over the past few years and today we're barely shy of 10 000 gpus which just to give you some sense in terms of number of gpu is more than the top five publicly known supercomputers in the world but that's not enough and so i'd like to invite ganesh to talk about the next steps [Applause]

1:45:34thank you milan my name is ganesh and i lead project dojo it's an honor to present this project on behalf of the multidisciplinary tesla team that is working on this project as you saw from milan there's an insatiable demand for speed as well as capacity for neural network training and elon prefetched this and a few years back he asked us to design a super fast training computer and that's how we started project dojo

1:46:12our goal is to achieve best ai training performance and support all these larger more complex models that andres team is dreaming of and be power efficient and cost effective at the same time so we thought about how to build this and we came up with a distributed compute architecture after all all the training computers out there are distributed computers in one form or the other they have compute elements in the box out here connected with some kind of network in this case it's a two-dimensional network but it could be any different network cpu gpu accelerators all of them have compute little memory and network but one thing which is common trend amongst this is it's easy

1:47:01to scale the compute it's very difficult to scale up bandwidth and extremely difficult to reduce latencies and you'll see how our design point catered to that how our philosophy addressed these aspects of traditional limits for dojo we envisioned a large compute plane filled with very robust compute elements backed with large pool of memory and interconnected with very high bandwidth and low latency fabric and in a 2d mesh format and on to this for extreme scale big neural networks will be partitioned and mapped to extract different parallelism model graph data parallelism and then in neural compiler of ours will exploit spatial and temporal locality such that it can

1:47:57reduce communication footprint to local zones and reduce global communication and if we do that our bandwidth utilization can keep scaling with the plane of compute that we desire out here we wanted to attack this all the way top to the bottom of the stack and remove any bottlenecks at any of these levels and let's start this journey in an inside out fashion starting with the chip as i described chips have compute elements our smallest entity or scale is called a training node and the choice of this node is very important to ensure seamless scaling if you go too small it will run fast but the overheads of synchronization will end software will dominate if you

1:48:50pick it too big it will have complexities in implementation in the real hardware and ultimately run into memory bottleneck issues because we wanted to address we wanted to address latency and bandwidth as our primary optimization point let's see how we went about doing this what we did was we picked the farthest distance a signal could traverse in a very clock very high clock cycle in this case two gigahertz plus and we drew a box around it this is the smallest latency that a signal can traverse one cycle at a very high frequency and then we filled up the box with wires to the brink this is the highest bandwidth you can feed the box with and then we added

1:49:37machine learning compute underneath and then a large pool of sram and last but not the least a programmable core to control and this gave us our high performance training node what this is is a 64-bit superscalar cpu optimized around matrix multiply units and vector cmd it supports floating point 32 b float 16 and a new format cfp8 configurable fp8

1:50:09and it is backed by one and a quarter megabyte of fast ecc protected sram and the low latency high bandwidth fabric that we designed this might be our smallest entity of scale but it packs a big punch more than one teraflop of compute in our smallest entity of scale so let's look at the architecture of this the computer architects out here may recognize this this is a pretty capable architecture as soon as you see this it is a super scalar in order cpu with four wide vector and two wide vect two wide four wide scaler and two wide vector pipes we call it in order although the vector and the scalar pipes can go out of order but for the purest out there we still call it in order and

1:51:00it also has four-way multi-threading this increases utilization because we could do compute and data transfers simultaneously and our custom isa which is the instruction set architecture is fully optimized for machine learning workloads it has features like transpose gather link traversals broadcast just to name a few and even in the physical realm we made it extremely modular such that we could start averting these training nodes in any direction and start forming the compute plane that we envisioned

1:51:39when we click together 354 of these training nodes we get our compute array it's capable of delivering teraflops of machine learning compute and of course the high bandwidth fabric that interconnects these and around this compute array we surrounded it with high speed low power services 576 of them to to enable us to have extreme i o bandwidth coming out of this chip just to give you a comparison point this is more than two times the bandwidth coming out of the state-of-the-art networking switch chips which are out there today and network switch chips are supposed to be the gold standards for i o bandwidth if we put all of it together we get

1:52:33training optimized chip our d1 chip this chip is manufactured in seven nanometer technology it packs 50 billion transistors in a miserly 645 millimeter square one thing you'll notice 100 of the area out here is going towards machine learning training and bandwidth there is no dark silicon there is no legacy support this is a pure machine learning machine and

1:53:07this is the d1 chip in a flip chip bga package this was entirely designed by tesla team internally all the way from the architecture to gds out and package this chip is like a gpu level compute with a cpu level flexibility and twice the network chip level i o bandwidth if i were to plot the i o bandwidth on the vertical scale versus teraflops of compute that is available in the state-of-the-art machine learning chips are there including some of the startups you can easily see why our design point excels beyond power now that we had this fundamental physical building block how to design the system around it let's see since d1 chips can seamlessly connect without any

1:54:13glue to each other we just started putting them together we just put 500 000 training nodes together to form our compute plane this is 1500 d1 chips seamlessly connected to each other and then we add dojo interface process processors on each end this is the host bridge to typical hosts in the data centers it's connected with pci gen4 on one side with a high bandwidth fabric to our compute plane the interface processors provide not only the host bridge but high bandwidth dram shared memory for the compute plane in addition the interface processors can also allow us to have a higher radix network connection in order to achieve this compute plane

1:55:18we had to come up with a new way of integrating these chips together and this is what we call as a training tile this is the unit of scale for our system this is a groundbreaking integration of 25 known good d1 dies onto a fan out wafer process tightly integrated such that it preserves the bandwidth between them the maximum bandwidth is preserved there and in addition we generated a connector a high bandwidth high density connector that preserves the bandwidth coming out of this training tile and this style gives us nine beta flaps of compute with a massive i o bandwidth coming out of it this perhaps is the biggest organic mcm in the chip industry multi-chip module

1:56:18it was not easy to design this there were no tools that existed all the tools were croaking even our compute cluster couldn't handle it we had to our engineers came up with different ways of solving this they created new methods to make this a reality now that we had our compute plane tile with high bandwidth ios we had to feed it with power and here we came up with a new way of feeding power vertically we created a custom voltage regulator module that could be reflowed directly directly onto this fan out wafer so what did we did out here is we got chip package and we brought pcb level technology of reflow on fan art wafer technology this is

1:57:12a lot of integration already out here but we didn't stop here we integrated the entire electrical thermal and mechanical pieces out here to form our training tile fully integrated interfacing with a 52 volt dc input it's unprecedented this is an amazing piece of engineering our compute plane is completely orthogonal to power supply and cooling that makes high bandwidth compute planes possible

1:57:59what it is is a nine petaflop training tile this becomes our unit of scale for our system and this is real

1:58:26i can't believe i'm holding nine petaflops out here and in fact last week we got our first functional training tile and on a limited limited cooling benchtop setup we got some networks running and i was told andre doesn't believe that we could run networks till we could run one of his creations this is min gpt2 running on dojo do you believe it

1:59:12next up how to form a compute cluster out of it by now you must have realized our modularity story is pretty strong we just put together some tiles we just styled together tiles a two by three tile in a tray makes our training matrix and two trays in a cabinet give 100 petaflops of compute did we stop here no we just integrated seamlessly we broke the cabinet walls we integrated these styles seamlessly all the way through preserving the bandwidth there is no bandwidth divert out here there is no bandwidth clips all the tiles are seamlessly connected with the same bandwidth and with this

2:00:12we have an exopod this is one extra flop of compute in 10 cabinets it's more than a million training nodes that you saw we paid meticulous attention to that training node and there are one million nodes out here with uniform bandwidth not just the hardware the software aspects are so important to ensure scaling and not every job requires a huge cluster so we plan for it right from the get go our compute plane can be subdivided can be partitioned into units called dojo processing unit a dpu consists of one or more d1 chips it also has our interface processor and one or more hosts and this can be scaled up or down as per the needs of any algorithm any network

2:01:23running on it what does the user have to do they have to change their scripts minimally and this is because of our strong compiler suite it takes care of fine-grained parallelism and mapping the pro problems of mapping the neural networks very efficiently onto our compute plane our compiler is uses multiple techniques to extract parallelism it can transform the networks to achieve not only fine-grained parallelism using data model graph parallelism techniques it also can do optimizations to reduce memory footprints one thing because of our high bandwidth nature of the fabric is enabled out here is model parallelism could not have been extended to the same

2:02:22level as what we can it was limited to chip boundaries now we can because of our high bandwidth we can extend it to training tiles and beyond thus large networks can be efficiently mapped here at low batch sizes and extract utilization and new levels of performance in addition our compiler is capable of handling high level dynamic control flows like loops if then else etc

2:02:52and our compiler engine is just part of our entire software suite the stack consists of a extension to pytarch that ensures the same user-level interfaces that ml scientists are used to and our compiler generates code on the fly such that it could be reused for subsequent execution it has a llvm backend that generates the binary for the hardware and this ensures we can create optimized code for the hardware without relying on even single line of handwritten kernel

2:03:32our driver stack takes care of the multi-host multi-partitioning that you saw a few slides back and then we also have profilers and debuggers in our software stack so with all this we integrated in a vertical fashion we broke the traditional barriers to scaling and that's how we got modularity up and down the stack to add to new levels of performance to sum it all this is what it will be it will be a fastest ai training computer for x the performance at the same cost 1.3 x better performance per watt that is energy saving and 5x smaller footprint this will be dojo computer

2:04:38we are not done we are assembling our first cabinets pretty soon and we have a whole next generation plan already we are thinking about 10x more with different aspects that we can do all the way from silicon to the system again we will have this journey again we're recruiting heavily for all of these areas thank you very much

2:05:14and next up elon will update us on what's beyond our vehicle feet fleet for ai [Music] oh [Music]

2:06:07so [Music] oh [Music] thank you [Applause] now unlike unlike dojo obviously that was not real so doj is real the tesla bot will be real but basically if you think about what we're doing right now with the cars tesla is arguably the world's biggest robotics company because our cars are like semi-sentient robots on wheels and with the full self-driving computer essentially the the inference engine on the car which will keep evolving obviously and dojo and all the neural nets recognizing the world understanding how to navigate through the world it it kind of makes sense to put that onto a humanoid form they're also quite good at sensors and batteries and

2:07:46actuators so we think we'll probably have a prototype sometime next year that basically looks like this and it's intended to be friendly of course and navigate through a world built for humans and eliminate dangerous repetitive and boring tasks we're setting it such that it is at a mechanical level at a physical level you can run away from it

2:08:26and and most likely overpower it so hopefully that doesn't ever happen but you never know so it's it'll be a you know a light yeah five miles an hour you can get run faster than that'd be fine [Laughter] so yes it's a round 5.8 has sort of a screen where the head is for useful information but as otherwise basically got the autopilot system in it so it's got cameras got eight cameras and

2:09:14full-size driving computer and making use of all of the same tools that we use in the car so i mean things i think that are really hard about having a useful humanoid robot is cannot navigate through the world without being explicitly trained i mean without explicit like line-by-line instructions can you can you talk to it and say you know please pick up that bolt and attach it to the car with that wrench and it should be able to do that it should be able to you know please you know please go to the store and get me the following groceries that kind of thing so yeah i think we can do that

2:10:03and yeah this i think will be quite quite profound because if you say like what is the economy it is at the foundation it is labor so what happens when there is you know no shortage of labor that's why i think long-term that there will need to be universal basic income yeah but but not right now because this robot doesn't work so we just didn't need a minute so yeah but i think essentially in the future physical work will be a choice if you want to do it you can but you won't need to do it and yeah i think it obviously has profound implications for the economy because given that the economy at its foundational level is labor i mean capital is capital equipment it's just

2:10:56distilled labor then is there any actual limit to the economy maybe not so yeah join our team and help build this all right so i think we'll we'll have everyone come back on the stage and you guys can ask questions if you'd like yeah [Applause]

2:11:37you [Music] all right there we go [Music] all right cool you guys brought me back on i just like stage try to get the camera angle from there but not from the side all right cool so we'll probably turn the lights back on and yeah great so we're having to answer any questions you have about anything on the software hardware side where things are going and yeah fire away we have because the lights are like interrogation lights so we actually cannot see ah there we go great

2:12:43all right cool i can just okay there we go first off i mean thanks to all the presenters that was just super cool to see everything i'm just curious at a high level and this is kind of a question for really anyone who wants to take it to what extent are you interested in publishing or open sourcing anything that you do for the future

2:13:27well i mean it is fundamentally extremely expensive to create the system so somehow that has to be paid for i'm not sure how to pay for it if it's fully open sourced yeah unless people want to work for free but but i should say that this is if other car companies want to license it and use it in their cars that would be cool this is not intended to be just limited to tesla cars

2:14:05is for the dojo supercomputer so did you solve the compiler problem of scaling to these many nodes or is or if it is solved is it only applicable to dojo because i'm doing research in deep learning accelerators and getting the correct scalability or the distribution even in one ship is extremely difficult from the research projects perspective so i was just curious excuse me mike for bill you have we solved the problem not yet are we confident we will solve the problem yes we have demonstrated networks on prototype hardware now we have models performance models showing the scaling the difficulty is as you said how do we keep the localities if we can do enough model parallel

2:14:57enough data parallel to keep most of the things local we just keep scaling we have to fit the parameters in our working set in our sram that we have and we flow through the pipe there's plenty of opportunities sorry as we get further scale for further processor nodes have more local memory memory trade also bandwidth we can do more things but as we see it now the applications that tesla has we see a clear path and our our modularity story means we can have different ratios different aspects created out of it i mean this is something that we chose for our applications internally

2:15:43sure the locality portion of it given that training is such a soft scaling application even though you have all this compute and have a high bandwidth high bandwidth interconnect it it could not give you that performance because you are doing computations on limited memory at different locations so i was that's very curious to me when you said it solved because i i just jumped onto the opportunity and would love to know more given that how much you can open source yeah yeah i guess the proof's in the pudding so we should have dojo operational next year and i think we'll we'll obviously use it for training video training it's i mean fundamentally this is about like

2:16:33the the primary application initially is we've got vast amounts of video and how we train vast amounts of video as efficiently as possible and also shorten the amount of time like if you're trying to train train to a task like just in general innovation is how many iterations and what is the average progress between each iteration and so if if you can reduce the time between iterations the rate of improvement is is much better so you know if it takes like sometimes a couple days for a model to train versus a couple hours that's a big deal but the the asset test here and you know what i've told the dojo team is like it's it's successful if the

2:17:24software team wants to turn off the gpu cluster but if they want to keep the gpp cluster on it's not successful so yeah hi right over here i love the presentation thank you for getting us out here loved everything especially the simulation part of the presentation i was wondering it looked very very very realistic are there any plans to maybe expand simulation to other parts of the company in any way

2:17:58hi i'm ian glow i manage the autopilot simulation team so as we go down the path to full self driving we're gonna have to simulate more and more of the vehicle currently we're simulating vehicle dynamics but we're going to need bms we're going to need the mcu we're going to need every single part of the vehicle integrated and that actually makes the autopilot simulator really useful for places outside of autopilot so i want to expand or we want to expand eventually to being a universal simulation platform but i think before that we're going to be spinning up a lot of optimist support and then a little bit further down the line that we have some rough ideas on

2:18:33potentially how to get the simulation infrastructure and some of the cool things we've built into the hands of people outside of the company optimus is the code name for the tesla bot oops optimus uprine [Laughter] hi this is ali jahanian thank you for the great presentation and putting all of these cool things together yeah for a while i have been thinking that the car is already a robot so why not a humanity robot and i'm so happy that today you mentioned that you are going to build such thing especially i think that this can give opportunity for rays of putting multi modality together for instance we know that in the example that you are showed that

2:19:31there was a dog and with some passengers or running together the language and symbolic processing can really help for visualizing that so i was wondering if i could hear a little more about this type of putting modalities together including language and vision because i have been working with for instance mini gpt's and andre put out there and yeah i had i didn't hear much about other modalities that's going into the car or at least in the simulation is is there any comment that you could tell us well driving is fundamentally basically almost entirely vision neural nets like we're basically it's running on a biological vision neural net and what we're doing here is

2:20:30a silicon camera neural net so and there are there's there is some amount of audio you know you want to hear if there's like emergency vehicles or you know i guess converse with the people in the car you know if somebody's yelling something at the at the car that car needs to understand what that is so you know all things that are necessary for it to be fully autonomous yeah thank you

2:21:11hi thank you for all the great work that you've shown my question is for the team because the data that was shown was seems to be predominantly from the united states that the the fsd computer is being trained on but as it is being as it gets rolled out to different countries which have their own road systems and challenges that come with it how do you think that it's going to scale like like i'm assuming like the ground up is not a very viable solution so how does it transfer to different countries well there is we actually do train on using data from probably like 50 different countries but we have to pick

2:21:53you know in as we're trying to advance full self driving we need to pick one country and since we're located here we pick the us and then we get a lot of questions like why not even canada like well because the roads are a little different in canada different enough and so when trying to solve a hard problem you want to say like okay what's the let's not add additional complexity right now let's just solve it for the us and then we'll extrapolate to the rest of the world but we do use video from all around the world yeah i think a lot of a lot of what we are building is very country agnostic fundamentally all the computer vision components and so on

2:22:29don't care too much about country specific sort of features every you know different countries have roads and they have curbs and they have cars and everything we're building is fairly general for that yeah and there's the the prime directive is don't crash right and that's true for every country yes this is the prime directive and even right now the car is pretty good at not crashing and so just basically whatever it is don't hit it even if it's a ufo that crash landed on the highway and still don't hit it you should not need to recognize it in order to not hit it so that's very important

2:23:20and i want to ask that when you do the photometric process multiview geometry how much of an error do you see is it like one millimeter one centimeter so i'm just if it's not confidential sorry question what is the what's it what's the difference between the synthetic sure what is the difference between the synthetically created geometry to the actual geometry yeah it's usually within a couple centimeters three or four centimeters that's the standard deviation merge with different kind of modalities to bring down that error we primarily try to find scalable ways to label in some occasions we use other sensors to help benchmark but we primarily use

2:24:10cameras for this system okay thanks yeah i i mean i think we want to aim for the car to be positioned accurately to the the sort of centimeter level you know something on that order obviously it will depend on distance like close by things can be much more accurate than farther away things because and they would matter less because the car doesn't have to make decisions much farther away and as it comes close it will become more and more accurate exactly a lot of questions thanks everybody my question has to do with sort of ai and manufacturing it's been a while since we've heard about the alien dreadnought concept is the humanoid that's behind you guys

2:24:50is that kind of brought out of the production health timeline and saying that humans are underrated in that process well sometimes like some you know something that i say is taken to too much of an extreme there there are parts of the tesla system that are almost completely automated and then there are some parts that are almost completely manual and if you were to walk through the whole production system you would see a very wide range from yeah like i said fully automatic to almost completely manual but the vast majority it's most of it is is already automated

2:25:33so and then with the some of the design architecture changes like going to large aluminum high pressure die cast components we can take the entire rear third of the car and cast it as a single piece and now we're going to do that the front third of the car is a single piece so the the body line drops by like 60 to 70 percent in size

2:26:01but yeah the the robot is not is not prompted by specifically by manufacturing needs it's it's just that we're just obviously making the pieces that are needed for a useful humanoid robot so i guess we probably should make it and if we don't someone else would well and so i guess we should make it and make sure it's safe

2:26:32i should say like also manufacturing volume manufacturing is extremely difficult and underrated and we've gotten pretty good at that it's also important for that humanoid robot like how do you make the human robot not be super expensive and hi thank you for the present presentation and my question will be about skilling of dojo and in particular how do you scale the compute nodes in terms of thermal thermals and power delivery because there is only so much heat that you can dispense and only so much power that you can bring to like cluster rack and how do you want to scale it and how do you point to scale it in multiple data centers sure

2:27:33hi i'm bill i one of the dojo engineers the so from a thermal standpoint and power standpoint we've designed it very modular so what you saw on the compute tile that will that will cool the entire tile so we we once we hook it up to it is liquid cooled on both the top and the bottom side it doesn't need anything else and so when we talk about clicking these together once we click it to power and we once we click it to cooling it will be fully powered and fully cooled and all of that is less than a cubic foot yeah so tesla has a lot of expertise in power electronics and in in cooling so we took the power electronics expertise from the vehicle powertrain

2:28:23and the sort of the advanced cooling that we developed for the power electronics and for the vehicle and applied that to the super computer because as you point out getting heat out is extremely important just really heat limited so yeah so it's funny that like at the compute level it's operating at less than a volt which is a very low voltage there's a lot of amps so therefore a lot of heat i squared r is what really bites you on the ass hi my question's also similarly a question of scaling so it seems like a natural consequence of using you know significantly faster training hardware is that you'd be either training models over a lot more data

2:29:10or you'd be training a lot more complex models which would be potentially significantly more expensive to run at inference time on the cars i guess i was wondering like if there was a plan to like also apply dojo as something that you'd be using like on the self-driving cars and if so like you know do you foresee additional challenges there i can so as you could see like android's models are not just for cars like there are auto labeling models there are other models that are like beyond car application but they feed into the car stack so so dojo will be used for all of those too not just the car inference part of the training yeah i mean the dojo's

2:30:02first application will be consuming video data for training for that would then be run in the inverse inference engine on the car but and that i think is an important test to see if it actually is good or but is it actually better than gpu cluster or not so but then beyond that it's basically a general a generalized neural net training computer but it's very much optimized to be a neural net so you know cpus and gpus they're they're they're not made to be they're not they're not designed specifically for training neural nets we've been able to make gpus especially very efficient for portraying neural nets but that's not that was never their design intent so

2:30:48it's it's bc gpus are still essentially running it neural net training in emulation mode so with with dojo we're saying like okay let's just let's just asic the whole thing let's just have this thing that's it's built for one purpose and that is neural net training and just generally any system that is designed for a specific purpose will be better than one that is designed for a general purpose hey i had a question here hi so you described two separate systems one was for vision therefore planner and control does dojo love you train networks that cross that boundary and second thing is if you were able to train such networks would you have the

2:31:30onboard compute capability in the fst system to be able to run that in in under your tight latency constraints thanks yeah i think we should be able to train planner networks on dojo or any gpus it's really invariant to the platform and i think if anything once we make this entire thing end to end it'll be more efficient than decoding a lot of these intermediate states so you should be able to run faster if you make the entire thing into in your neural networks we can avoid a lot of decoding of the intermediate states and only decode essential things required for driving the car yep certainly and to endness and as the guiding principle behind a lot of the

2:32:06network developments and over time in the stack neural networks have taken on more and more functionality and so we want everything to be trained end-to-end because we see that that works best but we are building it incrementally so right now the interface there is vector space and we are consuming it in the planner but nothing really fundamentally prevents you from actually taking features and eventually fine-tuning end-to-end so i think that's definitely where this is headed yeah and the discovery really is like what are the right architectures that we need to place in network blocks to make it amenable to the task so like on a describe we can place spatial rnns to

2:32:38help with the perception problem and now it's just neutral network so similarly for planning we need to bake in search and optimization into the planning into the network architecture and once we do that you should be able to do planning very quickly similar to c plus plus algorithms

2:32:58okay i think i had a question very similar to what he was asking about it seems like a lot of neural nets around computer vision and kind of traditional planning you had model predictive control in solving convex optimization problems very quickly and i'd wonder if there's a compute architecture that's more suited for convex optimization or the model predictive control solutions very quickly yeah 100 if you want to bake in like i said earlier if you want to bake in these architectures that do say model protein control but just like replace some of the blocks with neural networks or if we know the physics of it we can also use physics-based models part of

2:33:36the neural network's forward pass itself so we are going to go towards a hybrid system where we will have neural network blocks placed together with a physics-based blocks and more networks later so it'll be a hybrid stack and what we know to do well we place with explicitly and what the networks are created we'll use the networks to optimize this so better end-to-end stack with this architecture baked in i i mean i do think that so as long as you've got like surround video neural nets for understanding what's going on and can convert those surround video into vector space then you basically have a video game and if you know if you it's like if you're in grand theft auto

2:34:23whatever you can you can make the cars drive around and pedestrians walk around without crashing so you can do you don't have to have a neural net for control and planning but it's probably ultimately better so but i think you can probably get to in fact i'm sure you can get to much safer than human with control and planning primarily in c plus plus with perception vision in neural nets

2:34:56hi my question is we've seen other companies for example use reinforcement learning and machine learning to optimize power consumption and data centers and all kinds of other internal processes my question is are is tesla using machine learning within its manufacturing design or other engineering processes

2:35:19i i discourage use of machine learning because it's really difficult unless you basically unless you have to use machine learning don't do it it's usually a red flag when somebody says we want to use machine learning to solve this test i'm like that sounds like so 99.9 percent of time you do not need it so yeah but so it's kind of like a you you reach for machine learning when you when you need to not but it's i've not found it to be a convenient easy thing to do it's a super hard thing to do that may change if you've got a humanoid robot that can you know understand normal instructions but yeah generally minimize use of machine learning in the

2:36:16factory hi based on your videos from the simulator it looked like a combination of graphical and neural approaches i'm curious what the set of underlying techniques that are used for your simulator and specifically for neural rendering if you can share yeah so we're doing at the bottom of the stack it's just traditional game techniques just rasterization real time you know very similar to what you'd seen like gta on top of that we're doing real-time ray tracing and then those results were really hot off the press i mean we had that little asterisk at the bottom that that was from last night we're going into the neural rendering space we're trying out a bunch of

2:37:04different things we want to get to the point where the neural rendering is the the cherry on the top that pushes it to the point where the models will never be able to overfit on our simulator currently we're doing things similar to photorealism enhancement there's a paper a recent paper photo enhancing photo realism enhancement but we can do a lot more than what they could do in that paper because we have way more labeled data way more compute and also much we have a lot more control over environments and we also have a lot of people who can help us make this run at real time but we're going to try whatever we can do to get to the point where we can

2:37:40train everything just with the simulator if we had to but we will never have to because we have so much real world data that no one else has it's just to fill in the little gaps in the real world yeah i mean the simulator is very helpful when there's like these rare cases like like you know like collision avoidance right before an accident and then ironically that the better our cars become at avoiding accidents the fewer accidents there are so then our training set is small so then we have to make them crash in the simulation so it's like okay minimize potential injury to pedestrians and people in the car you have five meters you're traveling at

2:38:26you know 20 meters per second take what actions would minimize probability of injury we can run that in some cars driving down the wrong side of the highway that kind of thing happens occasionally but not that often for your humanoid contacts i'm wondering if you've decided on what use cases you're gonna start with and what the grand challenges are in that context to make this viable well i think for the humanoid for the tesla bot optimus it's basically going to start with just dealing with work that is boring repetitive and dangerous basically what is the work that people would least like to do

2:39:28hi so quick question about your simulations obviously they're not perfect right now so are you using any sort of domain adaptation techniques to basically bridge the gap between your simulated data and your actual real-world data because i imagine it's kind of dangerous to just deploy models which are solely trained on simulated data so maybe some sort of explicit domain adaptation or something is that going on anywhere in your pipeline so currently i mean we're producing the videos straight out of the simulator the the full clips of kinematics and everything and then we're just immediately training on them but it's not the entire data set it's just a

2:40:08small targeted segment and we only are evaluating based on real world video we're paying a lot of attention to make sure we don't ever fit and if we have to start doing fancier things we will but currently it's we're not having an issue with it overfitting on the simulator we will as we scale up the data and that's what we're hoping to use neural rendering to bridge that gap to push that even further out we've already done things where we're using like the same networks in the car but retrain it to detect sim versus rail to drive our art decisions and that's actually helped prevent some of these things as well yeah just to emphasize that overwhelmingly the data set is the real

2:40:46video from the cars on the actual roads nothing's weirder or or has more corner cases than reality it's gets really strange out there but but then if if we find say a few examples of something very odd and there's some very some for some very odd pictures we've seen then in order to train it effectively we want to create simulations say a thousand simulations that are that are variants of that quirky thing that we saw the foot to fill in the some important gaps and and make the system better and really all of this is about over time just reducing the probability of of a crash or an injury and it's called the march of nines like how do you get to 99.999999

2:41:36safe you know and it yeah each nine is an order of magnitude difficulty increase thanks so much for the presentation i was curious about the tesla bot specifically i'm wondering if there are any specific applications that you think the humanoid form factor lends itself to and then secondary because of its human form factor is emotion or companionship at all thought about on the product roadmap at all

2:42:18well we certainly hope this does not feature in a dystopian sci-fi movie but you know like really at this point we're saying like maybe this robot can just we're trying we're trying to be as literal as possible can it do boring dangerous repetitive jobs that people don't want to do and you know once you can have it do that then maybe you can do other things too but that's the that's the thing that we really great to have so it could be your buddy too i mean buy one and have it be or your friend and whatever i'm sure that people think of some very creative uses

2:43:08so so so firstly thanks for the the really incredible presentation my question is on the ai side so one thing we've been seeing is that with some of these language modeling ais we've seen that scaling has just had incredible impacts in their capabilities and what they're able to do so i was wondering whether you're seeing similar kinds of effects of scaling in in your neural networks in your applications absolutely a bigger network typically we see it performs better provided you have the data to also train it with and this is also what we see for ourselves definitely in the car we have some latency consideration to be mindful of and so there we have to get creative to

2:43:50actually deploy much much larger networks but as we mentioned we don't only train neural networks for what goes in the car we have these auto labeling pipelines that can utilize models of arbitrary size so in fact we've traded a number of models that are not deployable that are significantly larger and work much better because we want 100 like we want much higher accuracy for the auto labeling and so we've done a lot of that and there we definitely see this trend yeah the order labeling is an extremely important important part of this whole whole situation without the order labeling i think we would not be able to solve the self-driving problem it's kind of a

2:44:26funny form of distillation where you're using these very massive models plus the structure of the problem to do this reconstruction and then you distill that into neural networks that you deploy to the car but we basically have a lot of neural networks and a lot of tasks that are never intended to go into the car yeah and also as time goes on that you get new frames of information so you really want to make sure your computer is distributed across all the information as opposed to just taking a single frame and hogging on it for say 200 milliseconds you actually have newer frames coming in so you want to like use all of the information and not just use

2:44:56that one frame i think one of the things we're seeing is that the car's predictive ability is is quite is eerily good it's really getting better than human in terms of predicting like you say like what predict what this road will look like out when it's out of sight like it's around the bend and it predicts the road with very high accuracy and you know predict pedestrians or cyclists wherever behind you know where to just see a little corner of the bicycle and a little bit through through the windows of the bus and its ability to predict things is going to be much better than humans like really way way beyond right yeah we see this often where we have

2:45:41something that is not visible but the neural network is making up stuff that actually is very sensible sometimes it's eerily good and you have to like you're wondering this is in the training set and actually actually in the limit you can imagine the neural net has enough parameters to potentially remember earth so in the limit it could actually give you the correct answer and it's kind of like an hd map back baked into the weights of the neural net

2:46:08i have a question about the design of the tesla bot specifically in order how important is it to maintain that humanoid form to build hands with five fingers that also respects the weight limits could be quite challenging you might have to use cable driven and then that also causes all kinds of issues

2:46:30i mean this is just going to be bought version one and we'll see so the it's it needs to be able to do things that that people do and you know be a generalized you know humanoid robot i mean you could make you potentially have it give it like you know two fingers and a thumb or something like that you know for now we'll we'll give it five fingers and and see see if that works out okay it probably will it doesn't need to be like you know have like incredible grip strength but it needs to be able to work with tools so and you know carry a bag that kind of thing all right thanks a lot for the presentation so an old professor of mine told me that

2:47:21the thing he disliked a lot about his tesla was that the autopilot ux didn't really inspire much confidence in the system especially when like objects are spinning classifications are flickering i was wondering like even if you have a good self-driving system how are you working on convincing tesla owners other road users or other road users and just the general public that your system is safe and reliable well i think that's that's the cars a while back cars used to spend they don't they don't spin anymore not in the if you've seen the fsd beta videos they they're they're pretty solid and they will be getting more solid yeah as him add more and more data and train

2:48:02these multi-camera networks like these are pretty decent actually just like few months old and it's really improving it's not a done product and that we've never minds we can clearly see how this is just going to be like perfect but perfect for space because why not all the information is that in the videos it should produce a given lots of data and good architectures and this is just an intermediate point in the timeline i mean it's clearly headed to way better than human without question

2:48:33my turn oh hi here i was wondering if you could talk a little bit about the short to medium-term economics of the bot i guess i understand the long-term vision of replacing physical labor but i also think repetitive dangerous and boring tasks tend to not be so highly compensated and so i just don't see how to reproduce you know start with a supercar and then break into like the lower end of the market how do you do that for a robot humanoid well i guess you'll just have to see

2:49:16hello hi i was curious to know how the car ai prioritizes occupant safety versus pedestrian safety and what thought process goes into like deciding how to make this into the ai well i mean we the the thing to appreciate is that from the computer standpoint everything is moving slowly so think you know to a human things are moving fast to the computer they are not moving fast so i think this is in reality somewhat of a false dichotomy not that it will never happen but it will be very rare you know if you think it was like you know going the other direction like rendering you know with full ray tracing neural net enhanced graphics on something like cyberpunk or in any you

2:50:09know advanced video game you know doing 60 frames a second perfectly rendered like how long would it take a person to even render one frame and without any mistakes can be done i mean it would take like a month just to just render one one frame out of 60 in a second in a video game it's computers are fast and humans are slow

2:50:43i mean for example on on the rocket side the you you cannot steer the rocket to orbit we actually hooked up a joystick to see if anyone could steal the rocket orbit but you need to react at roughly six seven hertz people can't do it not even now that's pretty low you know we're talking more like even for like 30 hertz type of thing

2:51:15hi with the over here with hardware 3 there's been lots of speculation that with larger nets it's hitting the limits of what it can provide how much headroom has the extended compute modes provided at what point would hardware 4 be required if at all well i'm confident that hardware 3 or the full stop driving computer 1 will be able to achieve full self driving at a safety level much greater than a human probably i don't know at least two or three hundred percent better than a human then obviously there will be a future hardware for or full self-driving computer too which we'll probably introduce with the cybertruck so maybe in about a year or so

2:52:03that is probably well that'll be about four times more capable roughly but it's really just going to be like can we take it from say for argument's sake 300 safer than a person to a thousand percent safer you know just like there are people on the road who with with varying driving abilities but we still let people drive it you don't have to be the world's best driver to be on the road so as we see so yeah guess what's new all right are you worried at all since you don't have any depth sensors on the car that people might try like adversarial attacks like printed out photos or something to try to trick the rgb neural network yeah like what pull some like wiley cody

2:52:58stuff you know like paint the tunnel on the on the wall it's like oops we haven't really seen much of that i mean for sure like like right now if you pr most likely if you had like a a t-shirt with a t-shirt with like a stop sign on it which i actually have a t-shirt with a stop sign on it and and then you like flash the car it will it will stop i i proved that but we can obviously as we see these adversarial attacks then we can we train the cars to you know notice that well it's actually a person wearing a t-shirt the stop sign on it so it's probably not a real stop sign

2:53:54hi my question is about the prediction and the planning i'm curious how do you incorporate uncertainty into your you know planning algorithms do you just basically assume you know you mentioned that you run the the autopilot for all the other cars on the road do you assume that they're all going to follow those rules or are you accounting for the possibility that well they might be bad drivers for example yeah we do account for multi-modal futures it's not that we just choose one we account for this person can actually do many things and we use that actual physics and kinematics to make sure that they're not doing a thing that would interfere with us before we act so if

2:54:36there's any uncertainty we are conservative and then would yield to them of course there's a limit to this because if you're too conservative then it's probably not practical so at some point we have to assert and we even then we make sure that the other person can yield to us and act sensibly

2:54:54i should say like like before we introduce something into the fleet we will run it in shadow mode and so and we'll see what what would this neural net for example have done in this particular situation because and then effectively the drivers are training it training the net so if the neural net would have controlled and you know and say veered right but the person actually went left it's like oh there's a difference why was there that difference yeah and secondly all the human drivers are essentially training the neural net as to what is the correct course of action assuming it doesn't then end up in a crash you know doesn't count in that case

2:55:40yeah and secondly we have various estimates of uncertainty like flickr and when we observe this we actually say we are not able to see something we actually slow down the car to be again safe and get more information before acting yeah we don't want to be brazen and just go into something that we don't know about we only go into places where we know about yeah but yeah it should be like aspirationally that the car should be the less it knows the sl you know the slower it goes yeah which is not true at some point but now yeah yeah we've yeah should we speed proportionate to confidence

2:56:22thanks for the presentation so i am curious appreciate the fact that the fsd is improving but if you have the ability to improve one component along the ai stack to present it today whether it is simulation data collections planning control et cetera which one in your opinion is going to have the biggest impact for the performance of the full self driving system

2:56:51it's really the area under the curve of this like multiple points and if you improve anything it should improve the area i mean in the short term it's arguably we need all of the nets to be surround video and so we still have some legacy this is a very short term obviously we're fixing it fast but there's there's still some nets that are not using surround video and i think ideally that all use surround video yeah very yeah i think a lot of puzzle pieces are there for success we just need more strong people to also just help us make it work yeah that is the actual box so that is the actual bottleneck i would say i'm really one of the reasons that

2:57:30we are putting on this event exactly what well said andre that there's just a tremendous amount of work to do to make make it work so that's why we need talented people to join in and solve the problem thank you for the great presentation lots of my questions answered but one thing is when imagine that now you have a large amount of data even unnecessary how do you consider that like there's a forgetting problem in neural networks like how are you considering those aspects and also another one are you considering online learning or continuous learning so that maybe each driver can have their version of self-driving software i think i think i know the literature

2:58:27that you're referring to that's not some of the problems that we've seen and we haven't done too much continuous learning we train the system once we find in a few times that sort of goes into the car we need something stable that we can evaluate extensively and then we think that that's good and that goes into cars so we don't do too much learning on spot or continuous learning and don't face the forgetting problem but there will be settings that you can say like if you do do you want are you typically a conservative driver or do you want to drive fast or slow you know it's like i'm late for my i'm late for the airport could you go faster than you know

2:58:58basically the kind of instructions you'd give to your uber driver it's like i'm late for the flight please hurry or take it easy or you know whatever your style is so let's take a few more questions here so and then we'll call it a day all right so as our models have become more and more capable and i guess you're deploying these models into the real world one thing i guess that's possible is for ai to become more i guess misaligned with what humans desire so i guess is that something that you guys are worried about as you guys deploy more and more robots or do you guys like we'll solve that problem when we get there yeah i think that we should be worried

2:59:42about ai you know like what we're trying to do here is i say narrow ai pretty narrow like just make the car drive better than a human and then have the humanoid robot be able to do basic stuff you know so at the point at which you sort of start get superhuman intelligence yeah i don't know all bets are off

3:00:12but you know and that's that's that's you know that'll that'll probably happen but but what we're trying to do here at tesla is make useful ai that people love and and is unequivocally good that's our you know try to aim for that okay maybe one more question hi my question is about the camera sensor in the beginning of the talk you had mentioned about building a synthetic animal and if you think about it a camera is a very poor approximation of a human eye and a human eye does a lot more than take a sequence of frames have you looked into like there are like these days are like cameras like event cameras have you looked into them or are you looking into a more flexible camera

3:00:59design or building your own camera for example well with hardware four we will we will have a next generation camera but i have to say that the the current cameras we have not reached the limit of the current cameras so and i'm confident we can achieve full self-driving with much higher safety than humans with the current cameras and current compute hardware but you know are we good to be a thousand percent better rather than 300 better so we'll see continued evolution on on our levels and pursue that goal and i think in the future people will look back and say wow i can't believe we had to drive these cars ourselves you know it'll like self-driving cars

3:01:47will just be just a normal like self-driving elevators you know elevators used to have elevator operators and there's someone there with like you know big big relay switch operating the elevator and then every now and then they'd get tired or you know some make a mistake and share somebody in half so so now we you know we made elevators automatic and you just go and you press the button and you can be in a 100 story skyscraper and don't really worry about it just go and press a button and the elevator takes you where you want to go but it used to be that all elevators were operated manually manually it'll be the same thing like for cars all cars will be automatic

3:02:28and then and electric obviously so there will still be some gasoline cars and some manual cars just like there are still some horses so all right well thanks everyone for coming and i hope you enjoyed a presentation and thank you for the great questions [Applause] [Music] you

资源与出处

相关视频

同频道、同主题或相邻年份