I'll just throw this question out there:
Suppose we build an entirely new residence for the PM. Do you think it would be worthwhile to dismantle the current residence, move it elsewhere, and keep it as a heritage site?
The renovations would be minimal, merely to restore its structure. There wouldn't be any need to ensure it meets rigorous security needs or staffing functions.
Frankly, other than the fact that the house is old and a handful of prime ministers lived in it, it doesn't have much cultural value. Personally, I don't really find it worth keeping, even as a historical site.