From the impossible to the difficult to measure

It is important to know how people and society feel or how a product or service is experienced, and to be able to compare how these change over time and between different areas, for example, when making decisions about priorities, efforts and effects.

Sometimes expressions like “measuring the impossible” are heard when it comes to latent traits. Such expressions are not entirely true. Of course, latent traits are "hidden" and cannot be directly observed, but we can see signs of them through observable, so-called manifest expressions, such as perceptions or achievements. Imagine you are answering a questionnaire about how you feel, or you are taking an exam. The answers can say something about your well-being, just as your performance on the test can indicate an ability. Observing these manifest expressions is the first phase of a measurement, the so-called observation phase.

Interviews can be used to get preliminary knowledge and understanding of a latent trait. But if we want numbers—quantitative metrics—that can be used for calculations, comparisons and making decisions, we need to quantitatively measure the latent trait. How is your well-being, and how does it compare with others? How does your ability compare with the class, or for that matter the rest of the country?

Basic metrological principles

When many people say "measurement" in everyday speech, they often think of scores in a test or the percentage of people who answer in a certain way to a survey. Questionnaires, observation protocols or tests are often an important first basis for measuring latent traits, but a measurement also requires us to use basic measurement principles. When we use basic measurement principles to measure latent traits, which are based on data that we get in category form via questionnaires, observational protocols or tests, it is called category-based measurements.

One of the most important measurement principles for a measurement to be used for comparisons between different groups and over time is traceability to metrological references (so-called “standards”). For example, for the physical quantity “length”, traceability is relatively easy to understand. To know that your ruler has the right length, you can compare, calibrate it, against another standard ruler as a reference. The reference is in turn compared with other, more accurate references, and for complete comparability, all the way up to the primary realization of a metre as an international standard for the unit of length. Reliable and comparable measurements of latent traits need metrological references in a similar way.

From observations to measured values

When you answer a survey in the categories Can or Can't to a question, we only know that one category is “better” than the other, but not how big the difference between the answers is. How much more is Can than Can't? Often, "classification numbers" are given for estimates where, for example, the response category 0 (zero) corresponds to Can't and 1 (one) corresponds to Can. But there is no numerical value to the data we collected or the differences between them. This is called "ordinal" data and can be compared to a ruler where the scale lines have unknown distances between them.

The next step in a measurement process is the restitution phase, when we convert the ordinal observations, for example the survey responses or exam results, into linear, quantitative measurement values. Here, separate measures are given for two sets of latent traits: (i) measurement values for people's experience, feeling, behavior or ability and (ii) measurement value of the tasks. This separation is necessary for reliable and comparable measurement values as the measurement values of the tasks are used as metrological references. In category-based measurements, the so-called Rasch model is mainly used to do this. Of course, it's not as simple as just taking some data and putting it into the Rasch model. Category-based measures also require that we ask the right questions to the right people, and that the answers really provide observations for the latent trait we want to measure and not some other trait.

Doesn't it require more than that?

It is difficult to measure latent traits, but it is not impossible. Although the restitution phase is crucial for obtaining reliable and comparable measurement values, in many cases this is not done at all. Through the RISE initiative Center for category-based measurements, we work together with key actors to build and strengthen competence and develop methods and tools for category-based measurements, which turn the immeasurable into the difficult to measure. Although a more detailed description of how the Rasch model works is beyond the scope of this blog post, it is important to remember that there are models and tools to use.

One more thing to remember. If you have read this post, you have access to a computer and probably also a smartphone. We use these relatively new and advanced tools in our everyday lives, while most of us have very little understanding of how they function fully. Correspondingly, it is with measuring instruments: the ruler, scale or clock, etc. Even for measurements of latent traits, it might reasonably be the same way. Thus, not all users need to have a full understanding of how these new measurement tools work, but instead, a certain knowledge is needed of what is important to consider when enabling both reliable and comparable measurements of latent characteristics.

I hope that you partly gained new knowledge, partly that you have become curious to learn even more about going from the impossible to difficult-to-measure with category-based measurements and to reflect on how this can be done in your business. Do not hesitate to contact me or my colleagues for further discussion and dialogue about how you and your organization can be part of the continued journey!

RISE: The Swedish national metrology institute

As a national metrology institute, RISE is responsible through its national measurement sites for many of the physical quantities in the international system of measurement units, SI. This is part of an international measurement quality infrastructure that allows us to trust that a metre is a metre or a kilogram is a kilogram regardless of where in the world we are. The demands for increased credibility in measurements of latent characteristics means that we are also involved in driving the development towards a corresponding international measurement quality infrastructure for these characteristics as well.

RISE knowledge platform Center for category-based measurements is a Swedish initiative where RISE is expanding its role as a national metrology institute and builds and strengthens competence in measurements of experiences, feelings, behaviors and abilities