Physical custody doesn't matter either. What counts is a signed Form 8332 (assuming the divorce was within recent memory).