Photographing Houses in Garden District?

I was walking through the Garden District today (as a tourist, admittedly), and I was taking some photos of the neat houses as I passed by them. After just a couple, I got scolded by someone who looked like a lawn maintenance worker saying it was illegal in the state of Louisiana to photograph them and that I couldn’t be doing that. I think he said something about the reasoning being that children could be on the premises, but I couldn’t hear him that well over the lawn blowers.

I was on the sidewalk the whole time (and I wasn’t like zooming into rooms or anything—just taking general shots of the exteriors), so I thought what I was doing was perfectly fine, but I’m now wondering if I was breaking some law I didn’t know. I’m sticking around a few more days so would like to know what the deal is with this so I can be aware.