Cats, fun critters really.
Generally very honest about how energy flows around them. Very stealthy critters but very little guile to them.
I'd share your concern in that, the nature of the reaction seems like what you'd expect to a predator. One the cat doesn't know how to handle.
Incidents 2 yrs apart, was it the same house? Were the incidents coincident with changes/events in your life? Room mates? Seems unlikely whatever it is would "come for the cat". I mean, generally, people is much easier pickin's.
You were vocal, direct, and intent, so theres little doubt you were heard. Good move. "I don't know what frightened my cat", is probably the way to leave it, so long as they heard, and abide by your wishes. After all, striking up a relationship with whatever it was, just feeds the whole thing. You've made your position clear, and if it's limited to once every couple years something odd wanders through, well that's just luck of the draw.
After all, "it's a free country", right?
Fred