Yes you use the assets and liabilities of the estate to compute insolvency. But decedent's interest in the jointly owned property - house and mortgage- is part of his estate (just not part of the probate estate).
