Abstract

This chapter focuses on key psychometric and development issues related to creating and maintaining large-scale testing programs that require multiple versions of the same test. Large-scale testing programs give versions of the same test at different administrations but scores from each of the administrations must be comparable. Three broad areas of interest to test developers and statisticians are covered, namely, preliminary, operational, and maintenance requirements. Preliminary concepts that drive the ultimate design of the test are discussed, including use considerations (e.g., test purpose, target population, and criteria for score use), content considerations (e.g., content-related validity evidence, item format, and content specifications), and psychometric considerations (e.g., statistical specifications, scoring methods, choosing a reporting scale, and test length). Issues dealing with routine operational requirements for maintaining a viable testing program are also explored. Test delivery platform and security issues are discussed briefly. Greater attention is given to item and test development considerations, particularly considerations related to creating an optimal test inventory and determining the appropriate number of test items to write. Three basic methods for item tryouts are detailed and the pros and cons of each method described. Finally, requirements for maintaining a large-scale program are presented. Processes for building equivalent forms are described, test equating methods considered, and methods for maintaining the meaning of the scale discussed.