They are often the people would-be owner-occupiers have to outbid to buy a home, and the prices it is rational for them to pay are inflated by distortions in the taxation system, including the "expectation of high rates of return based on untaxed capital gains", in Spencer's words.
The problem arises from the different ways in which the tax system and the banking system see the purchase of an investment property.
The taxman says, "Congratulations on your new business. Here are all the deductions you can claim, including interest costs, incurred in earning this new taxable income, rent."
The banker does not see a business borrower, but someone wanting to borrow on the security of some real estate and is chiefly concerned with being able to get his money back in the event of a default. He will charge his lowest interest rate and happily lend as much of the purchase price as the regulator will allow. So the investor gets to enjoy all the advantages of leverage in a rising market until ready to sell up and pocket his untaxed capital gain.
It is a combination that ordinary business borrowers - the kind who employ people - and would-be owner-occupiers competing for the same property can only envy.
The broader case for a fresh look at the taxation of landlords' capital income is: (a) it is, after all, income and if you have an income tax you should tax income; and (b) that New Zealand is going to have to broaden its tax base to cope with the looming fiscal costs of an ageing population.
The last major review of the tax system, the tax working group chaired by Professor Bob Buckle in 2009, came down against a capital gains tax: "Most members of the tax working group have significant concerns over the practical challenges arising from a comprehensive CGT and the potential distortions and other efficiency implications that may arise from a partial CGT."
But they went on to say, "The other approach to base broadening is to identify gaps in the current system where income, in the broadest sense, is being derived and systematically under-taxed (such as returns from residential rental properties) and apply a more targeted approach. The majority of the [working group] support detailed consideration of taxing returns from capital invested in residential rental properties on the basis of a deemed notional return calculated using a risk-free rate."
This was a reference to the recommendation of an earlier review, headed by Rob McLeod, of an alternative way of taxing income from capital which it called the risk-free return method (RFRM).
The tax base under the RFRM is the amount that would have been earned if the funds invested in an asset subject to the regime had instead been invested in a risk-free government bond, and the portion of the return on the bond that represented compensation for inflation was exempt from tax.
The calculation would take the value of the asset at the start of the tax year, net off the outstanding principal on any related debt, multiply it by the inflation-adjusted risk-free rate of return and then apply the investor's marginal tax rate to that sum.
However RFRM was rejected out of hand by then Finance Minister Michael Cullen, and his successor Bill English ignored the Buckle review's call for a proper look at it in the case of residential rental properties.
And all we have got from Prime Minister John Key in response to Spencer's speech is the straw man argument that a capital gains tax is no prophylactic against house price inflation (as if that were the only option for taxing landlords' capital income). Look at Sydney. Look at London. They have a CGT and hot housing markets too. Trust the market. It will deliver the necessary supply side response eventually and it can be relied on to price dwellings in the secondary market accurately because we have not had a crash for 45 years.
Reassured? Me neither.
In any case, there is another argument for broadening the tax base which is demographic and fiscal.
In the current financial year the Government will spend $26.7 billion on New Zealand Superannuation and public health care. That is the equivalent of 11.2 per cent of gross domestic product or 41c in the dollar of its tax revenue.
But the Treasury has projected that by 2030 the Government's super and health care spending will have climbed to 14.1 per cent of GDP and 49c in the dollar of tax revenue, based on the ageing of the population and historic rates of cost growth.
Given our heavy reliance on a far from comprehensive income tax, those projections threaten one of the more useful features of New Zealand's tax system.
That is the relatively narrow tax "wedge" on wages. The tax wedge measures the difference between labour costs to the employer and the corresponding net take-home pay of the employee.
The OECD's annual report on this was released last week. New Zealand ranks at or near the bottom of the OECD rankings for the different categories of taxpayer considered.
The gap between what it costs to employ someone and his or her take-home pay is low by rich country standards, which means that tax is less of an impediment to employment.
For example, for a couple with two dependent children, where one partner is on the average wage and the other earning two-thirds of the average wage, the tax burden is 16.3 per cent compared with an OECD average of 31.3 per cent and 25.6 per cent in Australia.
This reflects the fact that New Zealand draws on the whole tax base to fund things like pensions and health care rather than relying on social security payroll taxes.
It is a strength of the New Zealand system but it is at risk the longer politicians shrink from taxing capital income properly.
