Fisher started competing in road and track races at age 12. He was suspended in 1968 because race organizers cited a rule that his hair was too long.

By 1972 this rule had been repealed and Fisher's career continued. He won the TransAlp race in Europe and a Masters XC national title. Fisher went to work in 1975 on his 1930s Schwinn Excelsior X bicycle.