According to history, the 4.84 acres of land where the State Museum now stands was granted by Raja Syed Alwi to his younger brother, Syed Salim bin Syed Muhammad al-Jafri, who was a senior minister to the King.

Towards the end of the 19th century, Syed Salim built his residence and stayed there with his family until it was sold to the State Government just after the signing of the 1909 Bangkok Treaty. Since then, the building became the official residence of the state British Advisors. The first British Advisor, Mr. Meadow Frost and his successors resided in the house until the Second World War in 1941. From 1941 until 1943, the building was taken over by Japanese Military Administration, while from 1943 until 1945, the Thai Military Government took over the building.

After Independence, the building became the Perlis Chief Minister's official residence. It became popularly known as Rumah Tetamu (Guest House) after it was vacated by Chief Minister Dato' Jaafar Hassan in early 1980s. In 1991, the building was demolished to make way for the construction of the present State Museum, which is structurally identical to the old building. The Kompleks Warisan Negeri, which is a new building in Kangar, is also within the compounds of the State Museum.