Even if you had an IR or cab simulation that perfectly replicated only the speaker itself, the FRFR cab, its dispersion pattern etc would make it sound different from what you hear in the room from the real guitar cab.
The moment you stop thinking of how to replicate the amp in the room experience, and treat miced cabs as just a different facet of all that, you will be happier. If I pipe my Fractal AM4 through cab sims my studio monitors, or direct into my BluGuitar Amp 1's poweramp and BluGuitar Fatcab 1x12 under my desk, I get two different sounds but not really a "this is better, this is worse" situation.