I haven't played with it much this year but it sure sounds like when it coughs up a hairball on these calculations it ends up with PTC and partial SEHI (double dipping).  I'm not sure that's the right answer if the math is unsolvable.

I'm wondering if our starting point for debugging these questions might be "what's the % of FPL on line 5 of the 8962?"

Just played around a little (I'm avoiding the PITA at the top of my stack).  I created a fringe case and it's giving me SEHI = the full APTC repayment even though the APTC repayment is limited.  In my case it was $2,850 SEHI but only $1,300 APTC repayment with 100% subsidy (so no additional out-of-pocket to be counted anywhere).  I'll have to look back at the rules for the non-iterative calculation but that's not my recollection of how it's supposed to work.

FYI, ProSeries considers this to be an "unsupported calculation."

Getting back to the OP, the general advice is to plug in some IRA and/or SEP-IRA contributions until the circular math equation becomes solvable.  We're running out of time on the IRA solution but an extension extends the SEP-IRA deadline until 10/15.

Rick