Population genetic models are fundamental to how we understand and infer the forces that patterned genome sequence variations in natural populations and across species. The most elemental model of population genetics is the discrete-time Wright-Fisher Markov model (WF), which mathematically describes the time evolution of genetic variation in idealized populations. WF is conveniently flexible and can account for a wide variety of important deterministic and stochastic forces, but it can be quite difficult to analyze. Often WF is studied by Monte Carlo methods, which can be time consuming and imprecise for rare events and random variables with broad distributions. Alternatively, WF can be studied mathematically using continuous-time diffusion approximations that were designed for convenient mathematics in an era that predated the wide availability of modern computers. In evolutionary genetics and molecular evolution, diffusion approaches remain in wide use, and are often deployed even when there is little compelling reason to do so (e.g., for models that lack closed-form solutions, requiring numerical integration than can be unstable and even misleading for unknown ranges of parameter choices). An important direction, now that we have access to both great data and expansive computational resources, is to evaluate the biological reasonableness of traditional simplifying assumptions, and to explore the consequences of relaxing those assumptions for how we understand fundamental evolutionary processes. In this dissertation, I build a comprehensive set of models and computational methods for rapidly and directly analyzing Markov chain models in population genetics without making any of the traditional simplifying assumptions like infinite sites, weak mutation, and weak selection. Our interest in these approaches was initially born out of a desire to better understand and model the population genetics of molecular evolution. However, it is my hope that this body of work will help stimulate renewed interest in Markov chain methods in population genetics in general, and that it may pave the way for more realistic, mechanistic, and tractable models of molecular evolution.