An interview with two experts about quality and criteria in learning materials
To start the new year 2022, the EEPG invited two experts to talk about quality in educational learning materials.
Andy Smart, education and publishing specialist and founder member of NISSEM.org; and Jim McCall, educational publishing consultant and BELMA jury member, talked to the director of the EEPG, Helga Holtkamp, about criteria that can be used in assessing educational materials. The complete talk is available for EEPG members but here is a summary in three questions.
H.H.: What are the basic requirements for producing quality learning materials and how can criteria for assessing materials be developed?
J.M.: The open market is an essential element in providing a reasonable range of choices for teachers when choosing new textbooks. That and a curriculum which can be relied upon to stay in place for a good period of time.
The important thing about evaluation criteria is that they offer an objective method of assessing quality in learning materials. This is much better than simply following the lead of other teachers. Criteria can be of use not just to Ministries of Education, approval committees and publishers but also to authors.
A.S.: The importance of choice is an element that runs through several parts of the curriculum chain. A curriculum framework can provide schools and teachers with a degree of choice as to what and how to teach, and a choice of textbooks is part of that freedom or autonomy. And this extends to the element of choice in how students learn in a classroom. Textbooks that do not allow any degree of teacher choice or student choice are more likely to result in rote teaching.
Where I would slightly differ from Jim is in the idea of the precision that textbook evaluation criteria can provide. To some extent, we can compare evaluating textbooks with marking exam papers, especially in subjects that involve some level of subjectivity, such as language and literature, or history. As with marking exams, the quality of the evaluation depends on the validity and reliability of the marking scheme as well as the process. I would say that sometimes, broad criteria work better than over-precise criteria. It depends on the culture of the system and whether the criteria are being used as a means of accountability or more for guidance.
H.H.: If publishers want to create their own criteria framework, what kind of standards have you come across and do they work?
J.M.: Where countries or rather Ministries of Education do have guidelines as to what they mean by 'quality', they tend to cover similar ground: issues on content, on pedagogical principles, and on design and presentation. Within these broad categories there are many sub-sets: 'content' would include adequate coverage of the curriculum, gender balance, and so on. Decisions about 'approval' may also have regard to price, but strictly speaking that is not a determinant of quality. I suppose an extreme example of 'standards' might be the old Soviet idea of 'state standards' which laid down rules for practically everything, from number of words to quality of paper.
A.S.: Those Soviet State Standards were (and sometimes still are) often known as technical and 'sanitary' (or safety) standards. They were based on often groundless ideas about measuring inputs in order to achieve good outputs. I think we can see something of this in the popular debate about 'what works' in teaching and learning. Dylan Wiliam, a well-known British education specialist now working mainly in the US, put it like this: 'Everything works somewhere; nothing works everywhere'. Even within a particular context it is difficult to show how particular features of a product are related to particular learning outcomes.
Getting the pedagogy right for the context is a good starting point for a high-quality materials. The challenge is then to turn a pedagogical approach into a set of valid criteria.
J.M.: Criteria can of course be selected to reflect the particular focus of individual subjects. The evaluation of history texts, for instance, will be rather different from those for maths or ELT. A focus on values, perhaps particularly on national as opposed to international values, will be important in this context. In more recent years, there has rightly been an emphasis on and an awareness of green issues, which was not always treated as well as it should have been.
H.H.: Do you have any suggestions as to how publishers can create criteria for their products and how they can make decision makers aware of the importance of quality publishing?
A.S.: One of the differences I see between professional educational authors on the one hand, and subject specialists (sometimes academics) as educational authors on the other hand, is that the professionals start with a pedagogical framework onto which they map the content of the syllabus. This pedagogical framework quite often features as a double-page spread at the start of a textbook, explaining 'How to use this book', which sometimes has arrows and captions. This is a good way to start - the publisher and author(s) can outline certain key points based on their own market research, following which the author can add the flesh to the bones.
These kinds of 'in house' criteria are useful as a way of aligning what the publisher wants (i.e. a product that will be successful in terms of adoption and selection, and which is distinct from the competition) with what the author believes will be an effective way of teaching the content.
J.M.: It would be useful to think about how ministries can be persuaded about the merits of private sector publishing as opposed to state publishing. Put simply, publishers have a skill set which Ministries of Education don't have, nor should they be expected to have. Ministry officials are not editors, and are not trained in or familiar with all the other aspects of the publishing process such as marketing and design and presentation.
If we consider 'readability' as an essential part of the mix, we naturally think about things like line length, typeface and type size, the balance of text and image, the use of white space, and the deployment of tints and icons and other visual devices to differentiate one kind of text from another, as well as the use of readable language. Publishers are familiar with all these issues and can apply that knowledge to the production of 'readable' learning materials, be they printed or online. It is at the very least unlikely that Ministries of Education, or 'textbook units' within them, will be able to call on that range of skills.
H.H.: Thank you both for this interesting exchange of ideas.