Poole Custom house was built in 1781 in Mid Georgian style. It was destroyed by fire in 1813 and replaced exactly as originally built. The original building was itself a rebuilding of the late 18th century Red Lion Coffee House. The striking entrance steps were copied from the Guildhall. The Custom House is part of a group of buildings that are all typical of an old quayside setting including the Town Cellar and Harbour Office.

The original Custom House building was built in 1747. In the same year, the Custom House was the site of a burglary, in which 30 smugglers, led by members of the Hawkhurst Gang, raided the house and took two tons of smuggled tea and 39 barrels of rum, worth over £500, that had previously been confiscated. The raid happened at night, with the gang reaching Poole at 11pm. The tea was then taken through Fording bridge. Four men were convicted at the Old Bailey in 1749, during which they argued that "there was no crime in smuggling" and that they were just "recovering their own goods". Three of the four men convicted were hanged at Tyburn.

The current Custom House in Poole was built in 1813, during a time in which Poole and Bournemouth were growing; the old Custom House building was destroyed by fire. In the nineteenth century, the Custom House was used for collecting extra taxes for boats entering Poole Harbour. In 1838, the building was run by Gartside & Co. The Custom House was closed by HM Treasury in 1883, with the customs being transferred to Weymouth's custom house. In 1954, the Custom House became a Grade II listed building.