Your selected text method is:
To document validation steps using the checklist, we recommend to download the Word Template below
Download (Word)tba
ValiTex - A Validation Framework for Computational Text-based Measures of Social Science Constructs
Beta Version 0.9
About
ValiTex is a novel validation framework for computational text-based measures of social science constructs (Birkenmaier et al., 2023). The framework offers practical guidance for researchers and users to conduct and communicate validation.
Overview
Validation is a critical task in text analysis and natural language processing. As its core, validation requires different activities to demonstrate that a method measures what it purports to measure (Cureton, 1951). However, validating text-based measures can be challenging (Krippendorf, 2018).
Therefore, any empirical measure needs to be validated. One crucial problem in the validation of text-based measures, however, is the lack of concepual clarity on how to conduct validation. To provide practical guidance for researchers and users to conduct and communicate validation, ValiTex provides a flexible and consistent appraoch to validation.
At its core, ValiTex requires three types of validation evidence: substantive, structural, and external evidence
-
Substantive Evidence:Requires to outline the theoretical underpinning of the measure. -
Structural Evidence:Requires to examine and evaluate properties of the model and its measures -
External Evidence:Requires to test for how the measure relates to other independent information or criteria
The framework is complemented by a checklist that defines and outlines empirical validation steps available to collect validity evidence for different use cases.
If you want to learn more about the framework and the checklist, please click on the respective section below or have a look at our Working Paper.
Framework
The framework is rooted in the well-established principles of measurement theory found within the psychometric literature which offers the most comprehensive and cohesive conception of validity for social science research. The complete framework is visually depicted in Figure 1.
Figure 1: Conceptual Model (click to expand)
Checklist
The dynamic checklist then provides a comprehensive list of validation steps for each phase within the structural model. Each validation step comes with a set of information, such as
| Variable | Information |
|---|---|
Status
|
Relevance of the validation step (either recommended or optional) |
Considerations
|
Detailed description of the validation step |
Reasoning
|
Information on how to evaluate the validation step |
Dimension
|
The type of validation step as defined in the conceptual model |
References
|
Reference to further literature and information |
To use the checklist in your research just click on Get Started in the navigation bar. There, you can generate your own checklist depending on the type of text method used. Furthermore, you will be able to download a Word template which you can fill out on your own and attach to your publication.
Figure 2: Screenshot Checklist (click to expand)
User Instructions
This application generates an adaptable checklist that you can use to validate your text-based measures. Each row within the table corresponds to one validation step (i.e., a single reported and clearly demarcated validation activity). Validation steps can be either recommended or optional depending on their relevance.
ValiTex accounts for different use cases in validation practices across text-based methods and research contexts. At present, ValiTex differentiates between seven use cases, which are highlighted in the Table below (click on it for a detailed view)
Table: Use Cases (click to expand)
Please start by choosing a method from the drop-down menu below. The complete list of validation steps is also available on Github.
Frequently asked questions
Here you will find answers to frequently asked questions about ValiTex If you have any questions that are not addressed here, please feel free to contact us! For a detailed description of the ValiTex framework, please refer to the working paper.
Feedback
Please note that this application is currently under construction and evolution. Please feel free to reach out to us with any feedback you have. We are eager to hear from you and will take your suggestions into consideration as we continue to develop and enhance our website. To reach us, please refer to lukas.birkenmaier@gesis.org
Citation
When refering to the site or the corresponding workin paper in publications, please cite the following references:
Birkenmaier, Lukas; Lechner, Clemens; Wagner, Claudia. (2023). ValiTex - a uniform validation framework for computational text-based measures of social science constructs. https://doi.org/10.48550/arXiv.2307.02863