Autonomous Vehicles (AVs) collect and pseudo-label terabytes of multi-modal data localized to HD maps during normal fleet tests. However, identifying interesting and safety critical scenarios from uncurated data streams is prohibitively time-consuming and error-prone. Although prior works have explored this problem in the context of structured queries and hand-crafted heuristics, we are hosting this challenge to solicit better end-to-end solutions to this important problem.
To this end, our benchmark includes 10,000 safety-critical natural language queries. Challenge participants can use all RGB frames, Lidar sweeps, HD Maps, and track annotations from the AV2 sensor dataset to find relevant actors in each log. Methods will be evaluated at three levels of spatial and temporal granularity. First, methods must determine if an action (defined by a natural language query) occurs in the log. Next, if the action occurs in the log, methods must temporally localize (e.g. find the start and end time) of the action. Lastly, methods must detect and track all objects relevant to the text description. Our primary evaluation metric is HOTA-Temporal, a tracking metric that jointly considers detection and association accuracy for referred objects.
Please see the Argoverse User Guide for detailed instructions on how to download the sensor dataset.
Please see our baselines to get started.
Please see the scenario-mining submission tutorial for a guide on preparing your submission.