Before the Orphan School opened in 1833, boys were housed in a disused distillery in New Town, and some girls placed in a private home in Davey Street, Hobart.
"Everyone knows how pleasing an appearance the exterior of the building exhibits; we wish we could say as much of the interior; but we cannot do, as the majority of the apartments allotted to the use of the children are cold comfortless, and ill arranged upon a most mistaken system of parsimonious economy...the prevalence of stone pavement, throughout the lower apartments of the building is, in our humble opinion, highly detrimental to the health of the inmates, in one room we saw five little fellows blue and shivering with cold, there was it is true a fireplace in the room, but no fire...We have seen many assemblages of children in our time...but never did we see two hundred human beings, that exhibited so squalid an appearance, as did the majority of the Queen's Orphans."
In 1841, well-known convict architect James Blackburn designed the two sandstone watch houses at the entrance to St John's Avenue. Both had an entrance porch. Blackburn's original plan for No.1 had two main rooms, one for men and one for women, and three smaller rooms. No.2 had three main rooms for constables.
Blackburn's watch houses may not have been the first on the site. Charles Bruce's etching in 1831 depicts two watchhouses in a similar location but there is no evidence that they were built.
"The reception of orphan children, children deserted by their parents, or the offspring of objects of charity who are unable to provide for them; the above classes are paid for by the Colonial Government. The other class and principally as regards numerical strength, being about six-sevenths, are those of convict parents undergoing probation or sentence, or illegitimate children of convict parents unprovided for, these are maintained at the expense of the Government, and may be said on average to cost £16 2s 3d per annum for each child (including all expenses), which charge includes the keeping of the buildings in repair."The Asylum also housed some Aboriginal children. It was believed at the time that although life there caused them 'distress and suffering', it also provided them with a knowledge of British culture and a level of education that assisted them in their struggle for integration into the growing white society.
The Infant Orphan School, constructed in 1862 became the Female Charitable Institution in 1874, operating as a lying-in hospital and home for "mentally defective" girls as well as providing accommodation for destitute women.