Farming was the primary occupation of the villagers in 1879, when land sold for a mere four dollars per acre. Surrounding villages were once more prosperous than Olney. Brookeville, just 2 miles to the north, had a population of 250. It served as the nation's capital for a day in August 1814, when the British invaded Washington, D.C. and James Madison fled the White House. To the west is Laytonsville, where 100 families lived. Both communities incorporated in the 1800s.