Text types in the Austrian Standardized National School-Leaving Exam for English An empirical study into the stability of text type constructs in assessment 1 Günther Sigott, Hermann Cesnik and Samuel Hafner At secondary level of the Austrian school system, great importance is put in the teaching and assessment of the ability to write texts that meet the conventions of specific text types. For the Austrian school system, descriptions of text types for German, English, French, Italian and Spanish are available. This study focuses on English. It builds on a study that investigated the nature of text type constructs which teachers hold when they are not looking at authentic writing performances from L2 learners of English in Austria (Sigott et al., 2024). The present study investigates the stability of teachers’ text type constructs under two different conditions. It compares teachers’ evaluations of authentic writing performances which were awarded top grades in the Austrian Matura Exam with the same teachers’ responses to a questionnaire that probed into the text type constructs which they held without looking at concrete writing performances. To do so, this study uses the questionnaire consisting of the 23 statements from Sigott et al. (2024). The teachers’ responses under the two conditions are compared for the text types essay, article, report, blog post, and blog comment. The results show that there is considerable stability in the teachers’ text type constructs under the two conditions. However, it also becomes evident that individual prompt formulations impact the nature of performances, and this is mirrored in the teachers’ evaluations. Overall, the results speak to the validity of the assessment of the Writing section of the Austrian Matura Exam. 1 This research was funded by Klagenfurt University, Austria. AAA - Arbeiten aus Anglistik und Amerikanistik Agenda: Advancing Anglophone Studies Band 50 · Heft 2 Gunter Narr Verlag Tübingen DOI 10.24053/ AAA-2025-0011 Günther Sigott, Hermann Cesnik and Samuel Hafner 168 1. Introduction At secondary level in the Austrian school system, but also in the school systems of other countries, it is considered important for students to be able to write texts that conform to the requirements of so-called text types. To a certain extent, this focus on text production integrates facets of genre-based language teaching (Hyland, 2018; Swales, 1990; Tardy, 2006, 2023; Yasuda, 2011) and text-based language teaching (Richards & Rodgers, 2022) into upper secondary language education. In the teaching of writing skills required in the Austrian school-leaving exam for English (Standardisierte Reife- und Diplomprüfung (SRDP)), students are exposed to definitions of text types for German and English. Students who choose a second foreign language are also exposed to such definitions for French, Italian or Spanish. However, the text types required for German are not aligned neatly with those which are required for the foreign languages English, French, Italian or Spanish. This is due to differences in teaching traditions for German versus those for the other languages. To some extent, this lack of correspondence is also brought about by differences in discourse conventions among the languages involved. The origins of text types for the SRDP are described in Struger (2018, p. 163) and Spöttl et al. (2018, p. 234). As a result, the students encounter different ways of defining text types for their language of instruction, which is German for the vast majority, and for their second and third languages. While the teaching supports them in understanding the salient features of individual text types within each individual language, the differences between the German text types on the one hand, and those for English, French, Italian or Spanish on the other, are rarely addressed explicitly, and students have to develop an understanding of these differences on their own. This challenge is heightened by the fact that the descriptions of text types provided by the Austrian Ministry of Education in the Text Type Characteristics (TTCs) (BMBWF, 2019) are often open to interpretation. In fact, due to the tension between the requirement of reliability of assessment and text authenticity this seems almost inevitable while it does make for uncertainty in the practice of teaching and for a lack of transparency in the practice of assessment. Thus, both teachers and students are faced with the challenge of forming their own understanding of how text types differ within and across the languages involved. In fact, attempts to study these understandings empirically have not been made until recently. The stability of these understandings is, however, an important factor in the validity of the assessment procedure for the writing part of the Austrian Matura exam. If assessors hold stable text type constructs regardless of the conditions under which they put them to use, this consti- Text types in the Austrian School-Leaving Exam for English 169 tutes an important contributory factor for the validity of the assessment procedure. Before teachers assess Matura writing performances, they must have formed a conceptualisation of each text type. This conceptualisation will function as the point of reference when they assess writing performances for these text types. These conceptualisations will also be the basis of writing instruction that prepares pupils for the Matura exam. In fact, one would hope that teachers’ conceptualisations of prototypical text types will be stable no matter whether teachers are asked about them directly, as happened in Sigott et al. (2024), or whether they use them to evaluate authentic writing performances. The actual text type constructs held by practicing teachers were empirically investigated and described in Phase 1 (Sigott et al. 2024), and recommendations for the practice of teaching and assessment were derived from the results. In that study, the text type constructs held by teachers were examined without teachers having access to concrete student performances. However, it cannot be taken for granted that teachers will react in the same way when they are asked about their intuitions concerning text types as they would when they look at actual text types. For instance, if teachers gave the text type ‘essay’ a rating of 2.9 on questionnaire item (statement) 10 “The text aims to shape opinion / wants to persuade”. in Phase 1, we would expect a similar value in Phase 2 (this study) for an essay performance which was awarded the top rating in the Matura exam. If this is the case, this would constitute support for the validity of the assessment procedure. Therefore, in Phase 2, the same data collection instruments were used, but this time teachers were required to answer with reference to authentic student performances from the Austrian national school leaving exam. In contrast to Sigott et al. (2024), where the teachers answered questionnaire items without access to writing performances in the current study, the teachers answered with reference to perceived properties of authentic performances. All of the performances had received top grades in the school leaving exam and can thus be considered prototypical examples of texts that fulfil the text type requirements for Matura. If the teachers maintain their text type constructs when evaluating authentic performances, the profile resulting from the questionnaire data in Phase 1 should be similar to the profile resulting from the analogous data in the present investigation. This study therefore addresses the following research questions: 1. In which aspects, if any, are there significant differences between the teachers’ conceptualisations of text type properties based on their intuition (Sigott et al. 2024) and their perception of text type properties of actual writing performances that received top ratings in the Austrian school leaving exam? Günther Sigott, Hermann Cesnik and Samuel Hafner 170 2. In the case of significant differences, what are possible reasons for these differences? 3. What do the results mean for the validity of the writing part of the Austrian school leaving exam? 2. Text types in the Austrian Standardised National School- Leaving Exam for English The Austrian curriculum for foreign languages in academic secondary schools (Allgemeinbildende Höhere Schulen (AHS)) as well as in colleges for higher vocational education (Berufsbildende Höhere Schulen (BHS)) is based on the CEFR. The target level for English, the first foreign language for the vast majority of students, is stipulated as B2. The SRDP - based on the curriculum and hence the CEFR - is a skills-based exam, consisting of three (BHS) or four (AHS) independent exam papers: listening, reading, language in use (AHS only), and writing. This study focuses only on the requirements for the writing part. Academic secondary schools generally focus on broad general education, whereas vocational schools offer a tailored professional education in a certain field. In the SRDP, the specific educational objectives of general and vocational schools are taken into consideration when it comes to assessing writing. Separate writing papers for AHS and BHS with differentiated concepts, and a particular focus on work-related topics in BHS, respond to the demands of the different school types. To be able to cover a range of functions, writing purposes and contexts, various text types are used in the examination. The current text types used in the SRDP are essay (for AHS only), article, e-mail, blog post, blog comment, report, and leaflet (for BHS only). The leaflet lends itself to testing action-oriented language use in work-related contexts and therefore enhances the face validity of the exam for BHS. The essay, on the other hand, requires the candidates to reflect on, and discuss, topics of general knowledge in a longer piece of writing as the set word length for essays is 400. Essay writing is thought to have more face validity in AHS with its focus on general education. Test-takers are required to write either two or three texts; one of 250 words and one of 400 words at AHS or three texts of 250 words each at BHS (see Table 1). Text types in the Austrian School-Leaving Exam for English 171 School types and text length Text type AHS BHS essay 400 N/ A article 250 / 400 250 blog post, blog comment 250 / 400 250 report 250 / 400 250 e-mail 250 250 leaflet N/ A 250 Table 1: Text types, school types and text length. Each test task consists of a prompt and instructions. The prompt defines the text type, sets the situational context, including an authentic situation, the role of the text-taker, the purpose of writing, the readership/ addressee and a stimulus (text-based or visual input). Three content points specifying required speech functions guide the test-takers through the writing task, as can be seen in Figure 1 and Figure 2. Fig. 1: Article for AHS, © BMBWF. The SRDP test papers are marked by the class teachers. A standardised analytic assessment scale was developed, consisting of four criteria (task achievement, coherence and cohesion, lexical and structural range, lexical and structural accuracy) and 10 bands, band 6 corresponding to the threshold for B2. The individual text type characteristics (TTCs) are outlined in the document „Übersicht Charakteristika Textsorten lebende Fremdsprachen (SRDP)“, which is available for students and teachers (BMBWF 2019). Detail on the individual text types is also provided in Sigott et al. (2024). Günther Sigott, Hermann Cesnik and Samuel Hafner 172 Fig. 2: Article for HLFS (Colleges of Agriculture and Forestry), © BMBWF. 3. Research design This study is the second phase of a two-phase project on text types. In the first phase, the focus was on practicing teachers’ text type knowledge (Sigott et al. 2024). In this second phase, the focus is on teachers’ perception of text type characteristics in authentic written student performances. The first study (Sigott et al., 2024) will henceforth be referred to as Phase 1, the present study as Phase 2. 3.1. Data collection method To tap into teachers’ perceptions of the text type characteristics embodied in the writing performances, the questionnaire from Phase 1 was used without alterations. This time the teachers were asked to characterise each concrete writing performance, rather than each text type in abstracto, from the Austrian SRDP for English by means of a 5-point Likert scale for each of the questionnaire items. The questionnaire, which is formulated in German, comprises 23 items (see Appendix, Table 4). In order to make the results comparable with a previous study on German (Sigott et al. 2020) and possible future studies for different languages in the school-leaving exam (French, Italian and Spanish), we decided to use the same statements in the same language, namely German. Translating could change the meaning of the items slightly, thus making Text types in the Austrian School-Leaving Exam for English 173 comparisons more difficult. However, after carrying out a content analysis of the TTCs, we added four statements (20 to 23) to ensure complete coverage of the TTCs in the questionnaire. The statements in their German version as well as their English translation can be found in the Appendix, Table 4. Respondents specified their level of consent to each statement on a symmetric agree-disagree scale, in analogy to the German study. Symmetry means that there is an equal number of positions around the ‘neutral’ value. The 5-point Likert scale ranges from ‘disagree’ (0), ‘partly disagree’ (1), ‘neutral’ (2), ‘partly agree’ (3) to ‘agree’ (4). The number of text types differs from the official number as described in the official text type characteristics, which specify 6 text types, namely essay, leaflet, article, report, blog, and e-mail. Due to the different discourse functions (statement/ question - reply), the blog post and the blog comment have come to be considered to be two separate text types in the practice of the SRDP, thus increasing the number of text types to choose from to 9. The present study is based on data for essay, article, report, blog post and blog comment. Data for leaflet and email, which would have fulfilled the criteria for inclusion, were not available. For performances from each of the 5 text types, namely essay (only AHS), article, report, blog post and blog comment, the teachers participating in the study were asked to respond to each of the 23 statements. 3.2. Participants In the present study, 25 of the 75 teachers who participated in Phase 1 were commissioned to assess two performances from each of five text types - ten performances per teacher - using the same questionnaire as in Phase 1 (see Appendix). The time required for this task was estimated at a total of 12 working hours, which constituted the basis for calculating the remuneration. All 75 teachers from Phase 1 were informed of the remuneration prior to recruitment. Ultimately, 25 practicing teachers (15 from AHS, 10 from BHS), based in the federal states of Salzburg (n = 9) and Carinthia (n = 16), chose to participate. As in Phase 1, the evaluations were conducted via an online survey. All 25 teachers completed the questionnaire in both phases, thereby allowing for a repeated measures design. Since only a subset of the original sample was involved in the present phase, the Phase 1 means reported here differ slightly from those in Sigott et al. (2024), which are based on the full cohort of 75 teachers. 3.3. Analysis The primary statistical measures used in this study are the sample mean and standard deviation (SD), which form the basis for paired-samples Günther Sigott, Hermann Cesnik and Samuel Hafner 174 t-tests. As in Phase 1, the mean is used to represent raters’ average agreement with the characterising statements. For each text type and each statement, the difference between the mean in Phase 1 (Mean (P1)) and the mean in the present study (Mean (P2)) is calculated (Δ Mean). All Δ Means were tested for statistical significance using paired-samples ttests. In addition, the p-value and effect size (Cohen’s d) are reported. Δ Means with an absolute smaller than 1 were not considered relevant for practical purposes, while a Δ Mean with an absolute of at least 1 was considered to be of practical relevance. All Δ Means that meet this requirement are also significant at p ≤ 0.00 (see Table 5 in the Appendix). 4. Results and discussion In Phase 1, all 23 statements were categorised on the basis of how well teachers agreed in their evaluations for each statement. The amount of agreement was used as a measure of the definingness of each statement. This yielded the categories of highly defining, defining and non-defining statements. 2 In the present study, those statements which turned out to be non-defining for all or individual text types in Phase 1 were no longer considered. Table 2 shows the mean for Phase 1 (Mean (P1)), the mean for Phase 2 (Mean (P2)) and those differences between the two means (Δ Mean) which are equal to or greater than 1 in absolute terms. The detailed statistics for all statements are given in the Appendix in Table 5. Text Type Statement Mean (P1) Mean (P2) Δ Mean Essay 20 The text makes comparisons. 3.00 1.47 1.53 Article 2 The text contains technical terms. 2.88 1.36 1.52 5 The text makes it clear to whom it is addressed. 2.20 1.20 1.00 11 The text is meant to entertain. 2.84 1.10 1.74 12 The text is narrative. 2.28 1.06 1.22 2 In Sigott et al. (2024: 51) this categorisation is described in detail: “The 23 statements in the questionnaire were put into three categories according to the amount of agreement reached by the teachers in indicating their consent. This agreement is here referred to as definingness. The definingness of statements is defined with regard to the values of the standard deviations. Statements with an SD smaller than 0.7 (SD < 0.7) are termed highly defining, those with an SD between 0.7 and 1.3 (0.7 ≤ SD ≤ 1.3) are referred to as defining, and those with an SD greater than 1.3 (SD > 1.3) are considered non-defining”. Text types in the Austrian School-Leaving Exam for English 175 19 The text summarizes / recapitulates. 2.72 1.12 1.60 Report 2 The text contains technical terms. 3.64 2.20 1.44 9 The text expresses an opinion. 1.16 2.18 -1.02 16 The text is appellative. 1.36 2.36 -1.00 20 The text makes comparisons. 2.80 1.18 1.62 Blog Post 2 The text contains technical terms. 1.16 2.26 -1.10 3 The text contains expressive or emotional expressions. 3.64 2.56 1.08 6 The text has a clear structure. 2.04 3.04 -1.00 11 The text is meant to entertain. 3.40 2.06 1.34 20 The text makes comparisons. 1.92 0.74 1.18 Blog Comment 3 The text contains expressive or emotional expressions. 3.68 1.90 1.78 6 The text has a clear structure. 1.88 3.50 -1.62 7 The text follows a common text pattern. 2.04 3.14 -1.10 11 The text is meant to entertain. 2.88 0.96 1.92 Table 2: Overview of Statements used for interpretation, grouped by text type. 3 4.1. Interpretation In a set of five text types - essay, article, report, blog post, and blog comment - a series of statements were rated first by teachers on the basis of their general expectations (Phase 1) and then by observing specific learner performances (Phase 2). Below is an overview of the key patterns and explanations for these discrepancies. All statements included here show differences of at least one full scale point (positive or negative) of 3 • The higher the mean, the higher the agreement to the statement. • Δ Mean represents the difference (Mean (P1) - Mean (P2)). Negative values indicate Phase 2 was higher than Phase 1. • Only statements whose absolute is equal to or greater than 1 are included. (|Δ Mean| = |Mean (P1) - Mean (P2)| ≥ 1). Günther Sigott, Hermann Cesnik and Samuel Hafner 176 the mean rating value. To compare the results of Phase 1 and Phase 2 visually, refer to Figure 3 in the Appendix. 4.2. Observations by text type A recurring reason for the observed divergences is the nature of the specific prompts. While teachers initially rated statements based on a “prototypical” conception of each text type (Phase 1), actual prompts often required different communicative functions (Phase 2). This created scenarios in which certain features (e.g., comparisons, narrative, entertainment) were either discouraged or outright irrelevant to the text’s intended purpose, leading to lower Phase 2 ratings than teachers had initially anticipated. Conversely, other prompts included requirements (such as explicit suggestions or technical vocabulary) that boosted certain features beyond prior expectations. 4.2.1. Essay The task requires a 400-word essay that argues for or against making a First Aid course compulsory for 14-year-olds. It should explain whether 14-year-olds would find this idea appealing, argue whether the state should fund the courses, and speculate on public reactions to 14-year-old first-aiders. Statement 20: the text makes comparisons (Phase 1 > Phase 2) The prompt appears to discourage candidates from engaging in explicit comparative analysis, as they are instructed to either support or oppose compulsory First Aid Courses. Moreover, the required communicative functions - namely, explanation, argumentation, and speculation - do not inherently promote comparative evaluation. Consequently, it is unsurprising that few comparative elements are evident in the performances. In fact, neither performance incorporates explicit comparisons. It appears that while teachers might expect a “good” or prototypical essay to make comparisons, not every essay prompt asks for them, and highly rated essays do not necessarily need to focus on comparing if the prompt does not call for it. 4.2.2. Article The task is to compose a 250-word article for an international online magazine aimed at helping students decide whether to spend a year abroad. The article must give reasons why practical experience is often more valuable than academic grades, describe the disadvantages of taking a gap year abroad, and suggest ways students can spend a year abroad. Statement 2: the text contains technical terms (Phase 1 > Phase 2) Text types in the Austrian School-Leaving Exam for English 177 The prompt is free of terminology that could be considered technical. It refers to school-related issues and matters closely tied to that context. The communicative functions required - giving reasons, describing, and suggesting - do not encourage, let alone necessitate, the use of technical language. It is therefore not surprising that the teachers rated the amount of technical language as relatively low in Phase 2. Statement 5: the text makes it clear to whom it is addressed (Phase 1 > Phase 2) The prompt places the focus on the argumentative content of the article rather than on the readership. The readership is broadly defined as “students”. This does not encourage candidates to define and address an audience more explicitly. Instead, the audience is construed as “students” or “young people”, and the magazine title, Working Teens, already defines the prospective readership beforehand, making it sufficiently clear to whom the article is addressed. Consequently, neither performance goes beyond addressing the readership with generic pronouns such as “you”, which leads to a rating in Phase 2 that is lower than anticipated by the teachers in Phase 1. Most likely, the need to make the readership of the article explicit is subdued by the formulation of the prompt, which explains the lower rating. Statement 11: the text is meant to entertain (Phase 1 > Phase 2) The prompt calls for a serious and argumentatively balanced text. The topic, spending a year abroad, addressed serious matters in the lives of young people, and while doubtless interesting for many, have little potential for being entertaining. As a result, the rating for whether the text is meant to entertain is considerably lower in Phase 2 than teachers’ initial expectations. Statement 12: the text is narrative (Phase 1 > Phase 2) The prompt clearly calls for an argumentative text that helps the readership, i.e., students to take a decision. This does not provide much room for narrative elements. If at all, then such elements could be supporting detail for arguments, but it would hardly be possible to accommodate them in a text of this length, particularly considering the three communicative functions that the text has to fulfil. Consequently, the rating in Phase 2 is noticeably lower than the Phase 1 rating. Statement 19: the text summarizes / recapitulates (Phase 1 > Phase 2) Because the prompt demands arguments (reasons, disadvantages, and suggestions) within a word limit of around 250 words, it is unlikely that candidates would add a separate summary or recapitulation. Integrating Günther Sigott, Hermann Cesnik and Samuel Hafner 178 elements of summarisation or recapitulation into the last section of the text would, given the shortness of the text, sound repetitive. Consequently, it comes as no surprise that the performances contain few, if any, elements of summarisation or recapitulation, explaining why the rating is much lower in Phase 2 than the relatively higher expectation from Phase 1. 4.2.3. Report The task involves writing a 250-word report for the headquarters of an international supermarket chain based on a customer survey conducted in the Austrian branch. The report should explain the reasons behind current issues, summarise the survey findings, and suggest measures to increase customer satisfaction. Statement 2: the text contains technical terms (Phase 1 > Phase 2) The prompt calls for a report from a supermarket chain’s customer service department to headquarters. Even though this is a business context, the needed terminology (e.g., staff, products, competitors) remains within common vocabulary, explaining the noticeably lower rating in Phase 2. Statement 9: the text expresses an opinion (Phase 1 < Phase 2) While teachers’ prototypical report focuses on factuality and objectivity and less on opinion, the performances reviewed in Phase 2 were rated higher for this statement. The reason might lie in the prompt since it calls for suggestions as one of the communicative functions. Suggestions inherently express a viewpoint which the performance contain to a certain degree. Statement 16: the text is appellative (Phase 1 < Phase 2) Similarly, requesting measures to increase customer satisfaction introduces an appellative (persuasive) element into the text, which again deviates from the typical notion of a neutral, purely factual report. Hence, the Phase 2 rating surpasses the lower Phase 1 expectation. Statement 20: the text makes comparisons (Phase 1 > Phase 2) The prompt does not specifically require comparing data sets but instead focuses on describing absolute values from the bar chart. Therefore, the potential for comparison never fully emerges, and the rating in Phase 2 is significantly lower than the more “typical” expectation teachers had in Phase 1. 4.2.4. Blog post The task involves writing a 250-word blog post to promote your awardwinning eco-shop as you set up an online shop. It should describe your Text types in the Austrian School-Leaving Exam for English 179 business idea, outline the challenges of setting up an eco-shop, and argue why online eco-shops are essential. Statement 2: the text contains technical terms (Phase 1 < Phase 2) The prompt itself references terms such as “Eco-Shop”, “promote your business”, “business community”, “blog”, “business idea”, and “online”. In both examined performances, additional vocabulary items considered “technical” appear, such as “eco-friendly”, “sustainable sources”, “online sector”, “suppliers”, “cost-efficient”, and “purchase”. which are relatively technical for a blog setting. This exceeds teachers’ initial, lower expectation for a “typical” blog post. Given the wide range of possible topics and communicative functions within any blog-prompt scenario, it is unsurprising that a concrete task focusing on eco-business strategies would yield performances that deviate from a more generic notion of blog writing. In this case, the heavier use of specialized vocabulary has led to a higher rating for this statement in Phase 2 compared to the lower Phase 1 expectation. Statement 3: the text contains expressive or emotional expressions (Phase 1 > Phase 2) Teachers initially expected a more expressive or emotional register, but the prompt’s main goals direct the text toward a factual style. While there are minor expressive cues (“I am happy”) and the use of punctuation (? , ! ) that could be seen as signalling expressiveness or emotionality, the overall emotional or expressive language falls below teachers’ Phase 1 expectations. Statement 6: the text has a clear structure (Phase 1 < Phase 2) The prompt requires attention to three clearly formulated aspects. The three performative verbs describe, outline and argue suggest a relatively stringent sequence of three paragraphs in the text. This structure is more rigorous than what teachers envisioned for an average blog post, so the Phase 2 rating surpasses the lower Phase 1 anticipation. Statement 11: the text is meant to entertain (Phase 1 > Phase 2) Because the prompt focuses on promoting a business and networking with a young business community, there is little need or room for an entertaining approach. Neither performance attempts to entertain per se; They do, however, attempt to attract the reader’s interest. Thus, the Phase 2 rating is significantly lower than the relatively higher Phase 1 expectation. Statement 20: the text makes comparisons (Phase 1 > Phase 2) Prompt does not explicitly, nor implicitly, ask for comparison; the three communicative functions - describing, outlining, arguing - do not call for Günther Sigott, Hermann Cesnik and Samuel Hafner 180 explicit comparisons. Consequently, neither performance presents comparative elements. Thus, the rating in Phase 2 is lower than the teachers’ moderate Phase 1 expectation for a “prototypical” blog post. 4.2.5. Blog comment The task involves writing a 250-word comment on a blog post that supports school security cameras. The text should outline school problems, discuss the impact of cameras, and suggest alternative solutions. Statement 3: the text contains expressive or emotional expressions (Phase 1 > Phase 2) The prompt asks for a discussion of a problem-solution text. This objective fosters a more factual tone with a suggestion of alternatives at the end, which reduces the likelihood of strong emotional or expressive language. Hence, the Phase 2 rating is notably lower than the high Phase 1 expectation. Statement 6: the text has a clear structure (Phase 1 < Phase 2) The prompt requires attention to three clearly formulated aspects. Both performances follow this pattern strictly by devoting a paragraph to each. The paragraphs are also clearly marked by layout. This gives the impression of a clearly structured text also to somebody who has not seen the prompt, as is true for the teachers evaluating the performances. Such systematic organization exceeds teachers’ idea of a “typical” blog comment, resulting in a higher Phase 2 rating compared to the lower Phase 1 rating. Statement 7: the text follows a common text pattern (Phase 1 < Phase 2) Like Statement 6, the text is shaped by the requirement to follow a standard “problem-impact-solution” pattern. Teachers, in Phase 1, expected a more free-form style, so the actual structured approach yields a higher rating in Phase 2 than they anticipated. Statement 11: the text is meant to entertain (Phase 1 > Phase 2) Again, a problem-solution approach centred on serious issues leaves almost no room for entertainment. The performances are factual and solutions-focused, standing in contrast to the more entertaining or anecdotal blog-comment style some teachers might envision, which leads to a much lower Phase 2 rating than predicted in Phase 1. Text types in the Austrian School-Leaving Exam for English 181 5. Conclusion This study investigated the extent to which teachers' conceptualisations of text type characteristics align with their evaluations of authentic student performances in the Austrian school-leaving exam for English. Addressing the first research question (In which aspects, if any, are there significant differences between the teachers’ conceptualisations of text type properties based on their intuition (Sigott et al. 2024) and their perception of text type properties of actual writing performances that received top ratings in the Austrian school leaving exam? ), 11 statements showed a significant difference in at least one text type. Some of the 11 statements showed a significant difference in more than one text type. An overview is provided in Table 3. Statement Significant difference in text type 2 The text contains technical terms. Report, Article, Blog Post 11 The text is meant to entertain. Blog Comment, Blog Post, Article 20 The text makes comparisons. Blog Post, Report, Essay 3 The text contains expressive or emotional expressions. Blog Comment, Blog Post 6 The text has a clear structure. Blog Comment, Blog Post 5 The text makes it clear to whom it is addressed. Article 7 The text follows a common pattern. Blog Comment 9 The text expresses an opinion. Report 12 The text is narrative. Article 16 The text is appellative. Report 19 The text summarises / recapitulates. Article Table 3: Statements with significant differences between Phase 1 and Phase 2. Regarding the second research question (In the case of significant differences, what are possible reasons for these differences? ), large discrepancies emerged across all text types whenever the prompt’s formulation prompted or constrained the text away from typical expectations. In some cases, teachers anticipated features that the tasks did not require (e.g., comparisons or entertainment). In other cases, tasks prompted unusual features for that genre (e.g., opinionated and appellative elements in a report). These findings indicate that large discrepancies between teachers’ theoretical expectations (Phase 1) and actual text performances (Phase 2) Günther Sigott, Hermann Cesnik and Samuel Hafner 182 often arise from the specific wording and communicative objectives of the prompts. It needs to be pointed out that there is a wide range of possible topics and communicative macro functions for prompts. Hence it is not surprising that a concrete prompt can give rise to performances which depart in some features from the ‘prototypical’ text type. As for the third research question (What do the results mean for the validity of the writing part of the Austrian school leaving exam? ), the implications for the validity of the writing component of the Austrian Matura exam are encouraging. From a broad perspective, out of 115 (23 x 5) 4 mean differences, 19 (16.5%) are large with an absolute delta mean of equal to or greater than one. Another 14 (12%) have smaller, but significant differences. Despite individual variations in prompt design and the subjectivity inherent in teacher assessment, the overall level of deviation is modest. Moreover, where discrepancies do occur, they are explainable through prompt-induced shifts rather than inconsistencies in construct understanding or assessment reliability. The findings therefore support the construct validity of the Matura writing assessment (or at least the sitting from which the performances were chosen) and highlight the importance of carefully designing prompts that guide learners toward producing text performances that align with the intended text type characteristics. Taken together, these results suggest that teacher conceptions of text types are relatively stable and reliable across abstract and performancebased evaluations, especially when task prompts are well-aligned with prototypical features. Future research could explore whether making prompt-specific communicative goals more explicit to both students and teachers can further reduce variance and support even greater transparency and fairness in assessment. 6. References BMBWF. (2019). Übersicht Charakteristika Textsorten lebende Fremdsprachen (SRDP): Stand August 2019. https: / / www.matura.gv.at/ downloads/ download/ uebersicht-charakteristikatextsorten-lebende-fremdsprachen Hyland, K. (2018). Genre and second language writing. In: J. I. Liontas (Ed.). The TESOL Encyclopedia of English Language Teaching. 2359-2364. Wiley Blackwell. https: / / doi.org/ 10.1002/ 9781118784235.eelt0535 Richards, J. C., & Rodgers, T. S. (2022). Text-based instruction. In: Approaches and Methods in Language Teaching. 200-214. Cambridge University Press. https: / / doi.org/ 10.1017/ 9781009024532.013 4 23 statements for each of 5 text types yields 115 mean differences for the entire data set. Text types in the Austrian School-Leaving Exam for English 183 Sigott, G., Krieg-Holz, U., Struger, J., & Cesnik, H. (2020). Was verstehen Lehrende unter Kommentar und Erörterung? Eine empirische Untersuchung zur Textsortenfrage in der österreichischen Standardisierten Reife und Diplomprüfung Deutsch. Der Deutschunterricht (5). 81-86. Sigott, G., Hafner, S., Cesnik, H., Weiler, T., Leitner, K., & Dousset-Ortner, E. (2024). Text types in the Austrian Standardized National School-Leaving Exam for English. Arbeiten Aus Anglistik Und Amerikanistik 49 (1). https: / / doi.org/ 10.24053/ aaa-2024-0002 Spöttl, C., Eberharter, K., Holzknecht, F., Kremmel, B., & Zehentner, M. (2018). Delivering reform in a high stakes context: from content-based assessment to communicative and competence-based assessment. In: G. Sigott (Ed.). Language Testing in Austria: Taking Stock/ Sprachtesten in Österreich: Eine Bestandsaufnahme. 219-239. Peter Lang. Struger, J. (2018). Deutsch als Unterrichtssprache: das Konzept der schriftlichen Reife- und Diplomprüfung. In: G. Sigott (Ed.). Language Testing in Austria: Taking Stock/ Sprachtesten in Österreich: Eine Bestandsaufnahme. 155-182. Peter Lang. Swales, J. (1990). Genre Analysis: English in Academic and Research Settings (13. Printing). The Cambridge Applied Linguistics Series. Cambridge Univ. Press. Tardy, C. M. (2006). Researching first and second language genre learning: a comparative review and a look ahead. Journal of Second Language Writing 15 (2): 79-101. https: / / doi.org/ 10.1016/ j.jslw.2006.04.003 Tardy, C. M. (2023). Genre‐based language teaching. In: C. A. Chapelle (Ed.). The Encyclopedia of Applied Linguistics. Wiley. https: / / doi-org/ 10.1002/ 9781405198431.wbeal0453.pub2 Yasuda, S. (2011). Genre-based tasks in foreign language writing: developing writers’ genre awareness, linguistic knowledge, and writing competence. Journal of Second Language Writing 20(2): 111-133. https: / / doi.org/ 10.1016/ j.jslw.2011.03.001 7. Appendix Original statement in German Translated statement (1) Der Text enthält explizite Markierungen von Verknüpfung, die durch Konjunktionen, Adverbien usw. angezeigt werden. (cohesion/ coherence) The text contains explicit markers of cohesion and coherence indicated by conjunctions, adverbs, etc. (2) Der Text enthält fachsprachliche Ausdrücke. The text contains technical terms. (3) Der Text enthält expressive oder emotionale Ausdrücke. The text contains expressive or emotional expressions. (4) Der Text ist für den öffentlichen Kommunikationsbereich bestimmt. The text is intended for public communication. (5) Der Text weist einen Adressatenbezug auf. / Der Text macht es klar, an wen er gerichtet ist. The text makes it clear to whom it is addressed. (6) Der Text weist eine klare Struktur auf. The text has a clear structure. Günther Sigott, Hermann Cesnik and Samuel Hafner 184 (7) Der Text folgt einem gängigen Textmuster. The text follows a common text pattern. (8) Der Text weist eine besondere stilistische Gestaltung und rhetorische Mittel auf. The text is written in a particular style and uses rhetorical devices. (9) Der Text äußert eine Meinung. The text expresses an opinion. (10) Der Text zielt auf Meinungsbildung ab / will von einem Standpunkt überzeugen. The text aims to shape opinion / wants to persuade. (11) Der Text soll unterhalten. The text is meant to entertain. (12) Der Text erzählt. The text is narrative. (13) Der Text erklärt. The text is explanatory. (14) Der Text argumentiert. The text is argumentative. (15) Der Text bewertet. The text is evaluative. (16) Der Text appelliert. The text is appellative. (17) Der Text beschreibt. The text is descriptive. (18) Der Text informiert. The text serves to inform. (19) Der Text rekapituliert/ resümiert. The text summarises / recapitulates. (20) Der Text vergleicht. The text makes comparisons. (21) Die Leserschaft wird direkt angesprochen. The reader is addressed directly. (22) Der Text soll Aufmerksamkeit erzeugen / auf sich ziehen. The text is meant to attract attention. (23) Der Text wurde geschrieben, um von einer größeren Leserschaft gelesen zu werden. The text is written to be read by a wider audience. Table 4: Questionnaire items (original and translation). Text types in the Austrian School-Leaving Exam for English 185 Fig. 2: Text type profiles. Note that the results for Phase 1 differ slightly from Sigott et al. (2024) since the data is based on the 25 teachers that participated in both studies Günther Sigott, Hermann Cesnik and Samuel Hafner 186 Notes for Table 5 (abbreviations): • M (P1) and M (P2) for Mean (P1) and Mean (P2) as used in text • Δ M for Δ Mean as used in text • C’s d for Cohen’s d from the paired samples t-test • P for probability value from the paired samples t-Test • SD (P1) and SD (P2) for standard deviations as used in text M (P1) SD (P1) M (P1) SD (P1) M (P1) SD (P1) M (P1) SD (P1) M (P1) SD (P1) M (P2) SD (P2) M (P2) SD (P2) M (P2) SD (P2) M (P2) SD (P2) M (P2) SD (P2) 3.67 0.62 3.60 0.58 3.44 0.87 2.40 1.15 2.32 1.38 3.33 0.77 2.82 0.80 2.72 0.80 2.94 0.87 3.24 0.68 3.00 1.00 2.88 1.09 3.64 0.64 1.16 0.80 1.24 1.01 2.33 0.94 1.36 1.18 2.20 1.17 2.26 0.99 1.28 1.06 1.47 1.06 2.32 1.18 0.64 1.04 3.64 0.49 3.68 0.56 1.60 0.71 1.36 0.80 0.78 0.72 2.56 0.93 1.90 0.66 2.47 1.06 3.88 0.33 1.80 1.35 3.76 0.52 3.68 0.56 2.83 1.16 3.02 1.08 0.70 0.94 3.96 0.14 3.30 0.90 0.87 0.92 2.20 1.08 3.24 1.09 2.80 1.04 3.64 0.57 0.57 1.07 1.20 1.32 3.66 0.62 2.90 1.14 3.94 0.22 3.53 0.64 3.44 0.58 3.96 0.20 2.04 1.21 1.88 1.20 3.10 0.83 3.00 0.82 3.00 0.82 3.04 0.80 3.50 0.41 3.07 0.96 3.00 1.12 3.68 0.85 2.16 1.21 2.04 1.14 3.27 0.88 2.78 0.83 2.78 0.66 2.90 1.04 3.14 0.87 2.93 1.03 2.76 1.05 2.64 1.15 2.16 1.34 2.16 1.18 2.27 1.32 1.90 0.97 1.80 1.07 2.76 0.98 1.88 0.97 2.93 0.80 2.44 1.16 1.16 1.03 3.80 0.41 4.00 0.00 3.33 0.75 2.08 1.15 2.18 0.90 2.92 0.81 3.74 0.48 2.93 1.10 2.60 1.22 1.40 1.22 3.16 1.03 3.60 0.71 2.90 0.60 1.96 0.99 2.30 1.05 2.94 0.89 3.28 0.52 1.47 0.92 2.84 1.11 0.28 0.46 3.40 0.87 2.88 1.01 0.90 1.04 1.10 0.97 0.16 0.31 2.06 1.17 0.96 0.90 0.93 0.80 2.28 1.10 0.60 0.96 3.00 1.12 2.40 0.87 1.20 1.18 1.06 1.07 0.84 0.87 2.78 0.99 2.20 0.98 3.33 0.82 2.72 0.94 3.56 0.65 2.52 0.82 2.64 0.86 3.20 0.68 2.90 0.54 3.32 0.68 3.28 0.58 2.96 0.69 3.93 0.26 2.92 0.95 2.48 1.19 2.28 1.14 3.32 0.80 3.47 0.72 2.86 0.99 2.86 1.06 2.12 1.00 3.10 0.91 2.80 1.15 2.28 0.94 2.08 1.55 2.56 1.00 3.32 0.80 2.33 1.13 1.70 1.11 2.70 1.05 1.52 0.84 2.76 0.83 2.67 1.18 2.56 1.16 1.36 1.11 2.76 0.97 2.76 1.05 2.20 1.08 1.94 1.01 2.36 1.21 2.30 0.75 2.38 0.78 2.87 1.06 3.12 0.67 3.40 0.96 2.84 0.94 2.68 0.95 2.57 0.98 2.66 0.67 3.14 0.65 3.12 0.63 2.90 0.75 3.27 0.70 3.56 0.65 3.76 0.52 2.84 0.75 2.36 0.81 3.00 0.96 3.04 0.69 3.66 0.37 3.46 0.59 2.42 0.66 3.00 1.07 2.72 1.10 3.12 1.09 2.12 0.88 2.44 1.08 2.67 1.03 1.12 1.09 3.00 0.68 1.34 0.81 2.00 0.94 3.00 0.85 2.28 1.17 2.80 1.22 1.92 0.91 2.36 1.04 1.47 1.26 1.40 1.05 1.18 0.89 0.74 0.74 1.40 0.91 0.73 0.46 2.68 1.18 1.20 1.12 3.56 0.77 3.32 0.99 0.67 1.16 2.14 1.09 1.88 1.06 3.88 0.42 3.02 1.08 2.47 1.13 3.56 0.82 1.36 1.15 3.52 0.65 3.40 0.65 2.13 1.11 2.38 1.05 1.54 1.08 3.70 0.72 2.38 1.00 2.27 0.96 3.76 0.44 1.20 1.04 3.68 0.75 3.32 0.90 2.60 1.24 3.16 0.83 0.38 0.53 3.96 0.14 2.80 0.95 C's d p report (n=25) Δ M C's d p blog post (n=25) Δ M C's d p article (n=25) Δ M C's d p 0.00 0.74 0.14 0.00 0.26 0.00 0.11 0.29 0.00 0.12 0.33 0.20 0.01 0.05 0.00 0.32 0.00 0.07 blog comment (n=25) Δ M 0.06 0.01 0.89 0.06 0.23 0.00 0.00 0.00 0.00 0.01 0.06 0.00 0.41 0.03 0.00 0.00 0.10 0.64 2.33 0.20 -0.33 0.38 -0.46 0.32 0.00 0.00 0.04 0.00 0.00 0.05 0.67 0.00 0.60 0.55 0.00 0.02 0.45 0.00 0.07 0.00 0.04 0.28 0.00 0.34 0.00 0.52 0.00 0.15 0.00 0.84 0.41 1.19 0.31 0.91 0.23 -1.26 -1.03 0.26 0.07 0.59 0.09 0.42 0.00 0.03 0.91 0.07 0.28 0.64 0.03 0.28 0.07 0.01 0.13 0.03 0.51 -0.29 0.12 0.61 0.29 0.33 0.25 -0.54 -0.03 2.21 0.97 0.17 1.37 0.22 0.63 0.54 -0.74 1.07 0.13 0.99 0.56 -0.32 -0.92 -0.04 0.00 0.15 0.00 0.00 0.23 -0.20 -0.07 0.41 -0.25 -0.69 0.39 -0.22 -0.06 0.38 0.20 0.06 0.00 0.00 0.00 0.22 -0.20 -0.10 -0.15 0.00 0.00 0.00 0.00 0.10 0.00 0.71 -0.43 -0.58 -0.39 -0.42 -0.09 -0.99 0.20 -0.70 1.34 -0.49 -0.29 -0.39 0.69 1.23 -0.34 0.76 1.14 1.24 -0.11 0.91 -0.65 0.25 1.32 -0.38 -0.78 -0.27 0.39 -0.38 0.69 -0.83 0.42 0.22 0.12 0.00 0.83 0.04 15 The text i s evaluative. 0.47 0.05 0.22 0.24 0.42 1.28 -0.28 0.88 1.07 0.74 0.28 0.65 0.46 0.16 -0.18 0.04 0.48 0.69 0.30 0.23 0.10 0.36 0.36 9 The text expres ses an opinion. -0.40 10 The text aims to shape opinion / wants to persuade. 0.03 13 The text i s expl anatory. 0.13 14 The text i s argumentative. 0.47 11 The text i s meant to entertain. 0.57 12 The text i s narrative. -0.27 0.52 0.30 1.02 0.44 0.96 -0.30 -1.62 1.78 0.38 1.92 0.20 0.26 0.32 -1.10 0.28 -1.10 1.08 0.82 -0.54 0.22 1.34 -0.60 0.88 -1.00 -0.74 0.46 -0.28 0.16 1.04 0.22 -0.76 -0.18 -0.28 1.18 -0.32 -0.62 0.78 0.72 1.44 0.73 -0.30 0.60 -1.02 -0.90 0.90 0.84 -0.42 0.96 -0.62 -1.00 0.24 -0.38 0.12 -0.24 -0.68 -0.18 0.12 1.62 0.26 0.10 1.55 1.10 0.38 1.18 -0.14 1.10 0.88 0.54 0.52 1.60 0.62 0.46 0.03 0.00 0.79 0.74 0.66 0.61 0.35 1.05 0.00 0.18 0.02 0.00 0.02 0.00 0.44 0.45 0.50 0.47 1.52 0.86 0.36 0.44 0.22 0.86 1.00 0.06 0.58 1.22 -0.18 0.64 1.74 0.96 0.21 0.51 -0.14 0.47 -0.45 0.05 0.67 0.18 0.42 23 The text i s written to be read by a wider audience. -0.33 0.78 21 The reader i s addres sed directly. 0.07 22 The text i s meant to attract attention. 0.33 7 The text follows a common text pattern. -0.20 8 The text i s written in a parti cul ar style and uses rhetori cal devi ces . 0.67 5 The text makes it clear to whom it i s addres sed. 0.30 6 The text has a clear structure. 0.43 19 The text summari ses / recapitul ates . 0.33 20 The text makes compari sons . 1.53 17 The text i s des criptive. 0.30 18 The text serves to inform. 0.27 16 The text i s appel ative. Statements 1 The text contains expli cit markers of cohesion and coherence indi cated by conjunctions , adverbs , etc. 0.33 2 The text contains techni cal terms . 0.67 3 The text contains expres sive or emotional expres sions . -0.13 4 The text i s intended for publi c communi cation. -0.37 essay (n=15) Δ M C's d p 0.32 0.55 -0.11 -0.36 0.24 Table 5: Statistics for statements per text type. Since the essay is only used in AHS, only AHS teachers (n=15) evaluated the essay.