That's just how RNG works. I had one character who drew the card the same day they got back from Irem, and another one who didn't get it for nearly a week.