Take reference data into account in memory usage tracking