This tool is developed for the general problem of comparing multidimensional simulation output with a given data set (e.g., real-world historical data). This problem frequently arises in verification, validation, and calibration of simulation models with spatial output statistics as in weather/climate, epidemic, swarm/crowd, social systems, communication networks, and many other applications where the simulation output is distributed across various locations or geographical regions. In the case of univariate simulation output, two-sample statistical hypothesis tests such as the t-test are commonly used. For simulation models with multidimensional and spatial output statistics, the Hotelling’s two-sample test is often recommended as the benchmark method in the simulation literature. However, the Hotelling’s test assumes that the two samples come from multivariate Gaussian distributions with equal covariance matrices, which may not be the case in many applications. To address this limitation, this tool provides a double-bootstrap method based on the Wasserstein distance for comparing two multi-dimensional samples. Unlike the Hotelling’s test and other parametric approaches, the proposed method does not require restrictive distributional assumptions, hence contributes to verification, validation, and calibration of simulation models with multidimensional output statistics.
Associated Research Article
A. Negahban (2025). “A Wasserstein Distance-based Double-Bootstrap Method for Comparing Spatial Simulation Output”, SIMULATION, under review.
Code & Sample Data
This code is a sample MATLAB implementation of the method proposed in the article referenced above. For the sake of simplicity and clarity, this sample code uses linear programming for computing the Wasserstein distance. The use of linear programming slows down the process (the example in this code takes about a minute to run). For faster execution time and higher dimensions, the method for computing the Wasserstein distance can be replaced with other methods.
ashkannegahban.com 