In this paper, we propose the Riemannian factor model, a novel framework for analyzing time series taking values in Riemannian manifolds in potentially high dimensions. Such time series is encountered in many applications, including economics, finance, medical imaging, and genomics and microbiome research. The proposed model is geometry-aware and accounts for the inherent nonlinearity in the data. Under a high-dimensional asymptotic regime, where the manifold dimension is allowed to diverge with $n$, the sample size, we establish convergence rates for the estimated loading space. In particular, under short-memory and strong factor conditions, we obtain a dimension-free $n^{-1/2}$ rate, which matches the convergence rate of the high-dimensional linear factor model. Applied to the realized covariances of selected U.S. stock returns, viewed as time series in the BuresWasserstein manifold, the proposed method yields interpretable factors and competitive predictions.