Well, if he is a noble it's just by a thread, since his family has been relegated to the lower-middle class due to debt and doesn't hold nearly the social and political influence they used to. A rivalry between their grandfathers might be interesting, though, since Ishimaru feels very strongly about him despite not knowing everything about the man. Knowing her vaguely from school is also viable. He might be a little apprehensive around her at first, since he doesn't know if she's a hard worker or not.

No need to feel silly! I'd be fine with whichever direction you want to take things in, but don't feel pressured to do a retcon if you don't want to.