When families like mine need to claim a piece of the affordable housing pie, those who need it most get even less.