A fundamental application of molecular simulations is to predict the atomic structure and self-assembly of materials. However, recent advances in experimental neutron and X-ray diffraction are revealing that many widely used force fields do not predict atomic structures that are consistent with experimental data. Directly reconstructing interatomic potentials from structure, the so-called inverse problem of statistical mechanics, is a possible method to improve and develop force fields for structure based applications. Here we present a machine learning assisted structure refinement technique, called structure optimized potential refinement, that can reconstruct transferable force fields from a single neutron diffraction measurement in monatomic fluids. Applying this technique to the noble gas series (Ne, Ar, Kr, and Xe), we recover transferable force field parameters that provide excellent agreement to experimental structure and reproduce vapor-liquid equilibria from the triple to critical point from a single neutron diffraction measurement. Furthermore, we show how structure optimized potential refinement can be used to quantify state dependent many-body interactions and quantum mechanical effects in fluid ensembles. These results underscore the importance of structure inversion techniques to numerous fundamental and interdisciplinary applications, including force field development, modeling fluids in extreme environments, and experimental quantification of quantum mechanical and many-body effects.