Fish that swim in schools benefit from increased vigilance, and improved predator recognition and assessment. Fish school size varies according to species and environmental conditions. In this study, we present a Hidden Markov Model (HMM) that we use to characterize fish schooling behavior in different sized schools, and explore how school size affects schooling behavior. We recorded the schooling behavior of Medaka (Oryzias latipes) and goldfish (Carassius auratus) using different numbers of individual fish (10–40), in a circular aquarium. Eight to ten 3 s video clips were extracted from the recordings for each group size. Schooling behavior was characterized by three variables: linear speed, angular speed, and Pearson coefficient. The values of the variables were categorized into two events each for linear and angular speed (high and low), and three events for the Pearson coefficient (high, medium, and low). Schooling behavior was then described as a sequence of 12 events ($\mathrm{<span\; style="font-family:\; 돋움;\; font-size:\; 11pt;">2</span>}<span\; style="font-family:\; 돋움;\; font-size:\; 11pt;">\times </span>\mathrm{<span\; style="font-family:\; 돋움;\; font-size:\; 11pt;">2</span>}<span\; style="font-family:\; 돋움;\; font-size:\; 11pt;">\times </span>\mathrm{<span\; style="font-family:\; 돋움;\; font-size:\; 11pt;">3</span>}$ ), which was input to an HMM as data for training the model. Comparisons of model output with observations of actual schooling behavior demonstrated that the HMM was successful in characterizing fish schooling behavior. We briefly discuss possible applications of the HMM for recognition of fish species in a school, and for developing bio-monitoring systems to determine water quality.