Tom was born in the back seat of a car on a snowy night in Eastern Wyoming, the youngest of a close-knit group of cousins and siblings, to his strong-willed mother and a Texas-born cowboy father. He spent his first seven years on the family's 14,000-acre ranch, where he learned the value of family, hard work, and the love of the land and nature.
His mother, Margaret Mary West, raised him to challenge himself and be strong in the face of difficulty, while his grandmother instilled in him a love for art and culture. He would spend hours looking at books of paintings and photographs, imagining himself as the characters he saw in exotic parts of the world.
At the age of seven, his family moved to New Mexico, where they had previously owned a ranch. His father, Byron Brown, continued to teach him to be a cowboy, while his mother taught him to be an artist. He took dance classes from his aunt, where he enjoyed being the only boy on a stage of girls, and studied classical dance for many years, learning the value of his studies as he grew as an actor.
When his father died of cancer four days after his ninth birthday, Tom turned inward and became angry about the loss, making it a difficult time for him and the family. He held on to his father's cowboy culture and was happy at the idea of moving to Montana to be on another big ranch. Before they were able to sell their small horse ranch, a film crew came to New Mexico to film Silverado, and his mother took him to an open casting call, where he got his big first break.
His family decided to move to California so he could pursue an acting career. Today, Tom continues to live between New Mexico, Montana, and California, and he and his wife raise their children with the same values they were brought up with.