If you finalized your divorce in 2004 you have to file either as a single or as a head of household. You should negotiate with your ex so each of you take one child as a dependent and the third would flip between you every other year. That way both of you can claim head of household (i.e. get lower tax rates) and you can share the dependent deduction and child tax credits.
Did you check your divorce decree papers to see if any of this was spelled out? Usually it is assuming it was handled by attorneys.