An algorithm learning tool was developed for an introductory computer science class in a specialized science and technology high school in Japan. The tool presents lessons and simple visualizations that aim to facilitate teaching and learning of fundamental algorithms. Written tests and an evaluation questionnaire were designed and implemented along with the learning tool among the participants. The tool’s effect on the learning performance of the students was examined. The differences of the two types of visualizations offered by the tool, one with more input and control options and the other with fewer options, were analyzed. Based on the evaluation questionnaire, the scales with which the tool can be assessed according to its usability and pedagogical effectiveness were identified. After using the algorithm learning tool there was an increase in the posttest scores of the students, and those who used the visualization with more input and control options had higher scores compared to those who used the one with limited options. The learning objectives used to evaluate the tool correlated with the test performance of the students. Properties comprised of learning objectives, algorithm visualization characteristics, and interface assessment are proposed to be incorporated in evaluating an algorithm learning tool for novice learners.