Poor whites didn't have rights. They made all kind of restrictions on voting. So person meant relatively well - off, free white man.