The former president was absent from court, but his counsel argued that health matters be left aside, so as to hear his case for the removal of Billy Downer