The rising interest in leveraging higher-order interactions present in complex systems hasled to a surge in more expressive models exploiting higher-order structures in the data,especially in topological deep learning (TDL), which designs neural networks on higher-order domains such as simplicial complexes. However, progress in this field is hinderedby the scarcity of datasets for benchmarking these architectures. To address this gap, weintroduce MANTRA, the first large-scale, diverse, and intrinsically higher-order dataset forbenchmarking higher-order models, comprising over 43,000 and 250,000 triangulationsof surfaces and three-dimensional manifolds, respectively. With MANTRA, we assessseveral graph- and simplicial complex-based models on three topological classificationtasks. We demonstrate that while simplicial complex-based neural networks generallyoutperform their graph-based counterparts in capturing simple topological invariants, theyalso struggle, suggesting a rethink of TDL. Thus, MANTRA serves as a benchmark forassessing and advancing topological methods, paving the way towards more effectivehigher-order models.