In Sylvia and Arthur LD's lifetime, Barrie did not have the boys staying with him. Andrew Birkin's book mentions (p 83) a letter that Sylvia LD wrote to Mary Barrie on 16 June 1901 about their arrangements for the summer holidays that year: "We have taken a charming cottage at Tilford, near Mr Barrie's, instead of Burpham...." Andrew B writes that "the cottage at Tilford was less than a five-minute walk along the dusty, winding road to Black Lake Cottage", but doesn't give the exact address. There is a picture in his book of Sylvia with George, Jack and Peter standing outside the cottage and there's another on this site's database of the cottage itself http://www.jmbarrie.co.uk/view/342/
As it's not mentioned again, one assumes the LD only rented it for that summer.
If you find out more about the actual address, and if the cottage is still there, do let this forum know.