The Effect of Written Feedback Types on Students' Academic Texts A Pilot Study Günther Sigott, Melanie Fleischhacker, Stephanie Sihler, Jennifer Steiner The aim of this pilot study is to examine how feedback type and error type influence the revision process of undergraduate writers. For the purpose of this study, text feedback was provided for eight essays, which were produced in a writing class for students who study English as a foreign language. Directive and non-directive feedback, differentiated by either providing or not providing students with suggestions for improvement, was distributed to approximately equal extents. The revised texts were then compared to the draft versions in order to analyse the students’ reactions to the feedback instances. Errors which triggered feedback were described in terms of the Scope - Substance taxonomy (Sigott, Cesnik, & Dobrić 2016). In this study, the focus was on Error Substance, i.e. the units of text that need to be modified, namely, text, paragraph, sentence, or word. The quantitative analysis of the feedback instances (n=90) has shown that written text feedback is generally highly effective, and that non-directive feedback leads to a more positive effect than directive feedback, particularly as far as lowerlevel errors (sentence or word) are concerned. The reasons for non-responses to feedback cannot be attributed to feedback type or error substance, which shows a distinct need for further qualitative analysis. AAA - Arbeiten aus Anglistik und Amerikanistik Band 44 (2019) · Heft 2 Gunter Narr Verlag Tübingen DOI 10.2357/ AAA-2019-0009 Günther Sigott, Melanie Fleischhacker, Stephanie Sihler, Jennifer Steiner 196 1. Introduction Teachers, lecturers, and writing tutors are often confronted with the challenges of giving constructive and encouraging text feedback on students' texts. Especially written text feedback is often neglected in writing instruction, even though it is a useful tool to engage with pupil and student writing. From an instructor’s perspective the practice of giving feedback itself frequently remains undefined and evasive. Most research in this area between 2010 and 2019 predominantly focused on hierarchy aspects between teachers and students or learners in general, perceptions of feedback practices (by teachers and students in the school and higher education context) as well as various forms of peer feedback and the benefits of developing peer feedback strategies (Ferris & al. 2011; Harran 2011; Young & Kwangsu 2011; Knorr 2012; Berggren 2013; Busse 2013; von Gunten 2015; Saliu-Abdulahi 2019). The contexts concerned are varied: Native English Speakers (NES), English as a Second Language (ESL), and English as a Foreign Language (EFL). In Germanspeaking contexts (especially Austria) research is missing regarding writers’ motivation to incorporate feedback during the revision process as well as on the factors that determine the effectiveness and pedagogical value of text feedback. This concerns native speaker contexts (school and higher education) as well as EFL contexts (school and higher education). Recent studies have neither focussed on giving examples of feedback instances (see Beyer 2018) nor provided comparison between drafts and revised text versions. Especially research on how writing tutors, who are neither teachers nor simply peers, give written text feedback with the aim of encouraging, supporting, and challenging feedback takers is still missing.Consequently, this pilot study explores the effects of written feedback types provided by professional writing tutors on academic papers written by EFL students at the University of Klagenfurt (Carinthia, Austria). The focus is on the effectiveness of directive versus non-directive feedback instances in the form of text commentaries. In this study, giving directive feedback has been defined as identifying the error in the text and providing suggestions for improvement, while non-directive feedback does the same without giving suggestions. Section 2 provides an overview of research concerning feedback giving and taking in various contexts. Section 3 contains the research questions addressed in this study. In Section 4, the participants, the data sample, and the research design are introduced. In Section 5, the most relevant results are presented and discussed. Finally, the conclusion focuses on implications and further research possibilities. The Effect of Written Feedback Types on Students' Academic Texts 197 2. Background Research in this area has been carried out in different geographical contexts (United States, Europe, Asia), educational institutions (different school forms and higher education), educational contexts (NES, ESL, and EFL contexts), and has observed feedback practices of different groups of individuals (teacher-student, or student-pupil feedback vs. peer feedback). Furthermore, different forms of feedback (i.e. marginal comments, shorthand corrections, end comments) have been investigated. US researchers started being interested in feedback practices (mainly teacher response in higher education) as early as in the 1970s (Jamalinesari & al. 2015: 116). Notable first articles concerning NES student writing include Sommers’s (1982) and Brannon and Knoblauch’s (1982). Sommers’s observations are among the first concrete engagements with teacher commentary in the US context. Her study is concerned with the effectiveness of teacher comments, but also tries to trace teachers’ motives in commenting on student texts. She points out that teachers’ pedagogical aims might interfere with the effectiveness of their feedback. They may inadvertently deprive writers of their own voices, urging them to alter their texts to fit the teachers’ ideas rather than upholding their own purpose. Also, teacher comments are often overgeneralised and vague. In addition, Sommers observes that teachers tend to adopt a product rather than a process-oriented approach, focussing on low-level errors first, or conflating comments concerning both higherorder and lower-order concerns. Feedback might even be contradictory. Text revision is therefore not experienced as a process, but as a single, little rewarding activity (151) and students tend to change only what is asked of them (152). Sommers suggests that teachers should adapt their comments to the respective draft (155), rather than treating drafts as final text versions (145). Comments should not remain “disembodied” (155), but teachers need to come back to them in different classroom situations, thus “framing revision as a learning process” (ibid.). Brannon and Knoblauch (1982), who share their data set with Sommers, also warn against the danger of feedback givers appropriating students’ texts rather than letting them define goals for their own writing. Straub (1996) is particularly interested in varying degrees of control that teachers (may) exert over student writing via feedback. In a subsequent study (Straub 1997), he investigates the reactions of students to teacher commentary using a questionnaire, asking students to comment on focus, specificity, and the mode of the individual comments. He found that students prefer elaborate and specific comments, but dislike criticism of their ideas. Connors and Lunsford (1993) looked at a combination of marginal, interlinear, initial and end comments by teachers, with end responses being the most commonly used form of written response in the 3000 papers that were analysed. They initially wanted to limit their anal- Günther Sigott, Melanie Fleischhacker, Stephanie Sihler, Jennifer Steiner 198 ysis to “global com-ments”, which they defined as comments that focus on large-scale issues like text structure, use of genre conventions and content rather than mere error correction (205). But they soon discovered that comments referring to grammatical errors were frequent, so they expanded their focus to these as well (206). Their aim was to analyse how teachers “judge the rhetorical effectiveness” (206) of student texts and what kind of teacher-student relationships could be derived from the comments (ibid.). Connors and Lunsford tried to identify “tropes” (210) (i.e. patterns) within the commentaries and to relate their rhetorical form to their purpose (ibid.). They identified four tropes: positive comments (rare but when given very personal), admonition comments (responses which first criticise mechanical aspects of a text and then move into a more positive comment on the effectiveness of certain text aspects), comments that start positively and end on a negative note and exclusively negative comments (ibid.). The most commonly used responses were of the positive-negative variety, usually starting out with positive feedback on a rhetorical element of the text (higher order concern) and leading towards negative feedback on a mechanical issue (lower order concern) (211). The authors refer to the length and form of the comments briefly (211), but they do not provide actual examples of specific comments and how the feedback interacted with the student texts. The length of the comments however, seems to be associated with the kind of problems addressed. Shorter comments are used to address lower-level errors and longer comments to address higher-level errors. Longer comments are rare, however, and teachers do not seem to be concerned with students’ further development. Initial and end comments are used primarily to explain the grade (211-213). Additionally, some teachers assume an overly authoritarian position, formulating ill-fitting, because insensitive and directive, comments (215). Overall, Connors and Lunsford judge the comments as grade-driven (217), meaning that teachers tend to ignore the rhetorical contexts of their interventions, choosing to follow a product-oriented approach to engage with their students’ texts (219). Research into the ESL context in the US remained scarce for a long time. Zamel’s (1985) study is one of the earliest engagements within this context. Her article challenges notions of product-oriented writing instruction and explores differences between L1 and ESL settings. She finds that teacher feedback tends to be unspecific, contradictory and difficult to decipher for ESL students. Furthermore, ESL teachers tend to regard themselves as language teachers rather than writing instructors (86). Therefore, they focus on surface-level errors even more severely than L1 teachers. This limits students’ understanding of writing and revising and hardly leads to any text improvement (79). Zamel analyses short text passages, which include the teachers’ comments and the resulting revised text, and isolated text passages. Her text data shows that the teachers’ generic approaches to feedback giving may stifle the students’ develop- The Effect of Written Feedback Types on Students' Academic Texts 199 ment and that teachers often misinterpret students’ intentions (86-93). Reid (1994) looks at the difference between intervention and appropriation expressed in teacher comments. While appropriation, i.e. the feedback giver taking control of the text, should not be the goal, constructive intervention, i.e. feedback that preserves and respects the writer’s voice, might be important to prepare students for effectively communicating in new discourse communities. Hyland and Hyland (2001) focus on the ESL context and teacher feedback in the form of praise, criticism, and suggestions. They mainly look at end comments. Ferris (1997) is one of the few researchers who are concerned with the linguistic features of teacher comments as well as with the linguistic aims expressed by them. Her study examines marginal and end comments given on the texts of advanced ESL learners, focussing on the linguistic features and the pragmatic goals of the comments as well as the revisions made based on these comments. Ferris focuses on various comment features, “including their length (in number of words), their type (pragmatic intent and syntactic form), the use of hedges (e.g., please, maybe), and whether the comment was text based or general” (320). Comments led to either positive revisions or to no change at all. Questions and statements that provided information were overall not helpful for students. Ferris does not give possible reasons for students ignoring feedback, however. Ferris et al. (2011) focus not on “the quality or effect of teacher feedback but rather [on providing] an objective and consistent quantitative description across a range of teachers of their general approaches to written feedback” (217). Similarly, Ferris (2014) deals with teacher’s philosophies underlying their feedback and their feedback practices. Conrad and Goldstein (1999) look at comments, revisions and individual and situational characteristics that may have influenced the revision process of ESL students. They examined comments by one teacher given on texts that do not all qualify as academic writing. The feedback was separated from the texts and given on a separate sheet of paper. The researchers observed that students found comments related to text development (adding examples, facts and details but also developing an argument, expanding on explanations, making analyses more explicit) difficult to react to. The characteristics of the comments did not influence revisions consistently, while individual factors did quite strikingly. Individual factors were determined in teacherstudent conferences that were recorded and analysed. Further research is concerned with teacher-student conferences, revision practices, peer feedback practices, or corrective feedback. Patthey- Chavez and Ferris (1997) explore writing conferences and how students’ proficiency levels and their cultural and personal backgrounds (status differences) may shape them. They compare ESL and NES conferences. Fitzgerald (1987) summarises research in revision studies. Tschudi, Estrem, and Hanlon (1997) are equally concerned with helping students develop effective revision strategies. Nelson and Shunn (2009) focus on Günther Sigott, Melanie Fleischhacker, Stephanie Sihler, Jennifer Steiner 200 peer feedback in a native speaker context. Kwangsu, Schunn, and Wilson (2006) compare strategies used in peer feedback with those used by teachers. Young and Kwangsu (2011) present what they refer to as the “learning-writing-by-reviewing hypothesis” (630). Their aim was to determine what effects giving comments might have on the peer reviewers’ writing skills. However, they did not explicitly focus on how writers use comments to actually revise their texts. Leki (1990), Liu and Hansen (2002), and Connor and Asenavage (1994) observe peer feedback strategies among ESL writers at university level. Leki (1991) is also concerned with corrective feedback (foremost concerning grammar issues) in US ESL classrooms, while Brown (2012) focuses on native speakers. Jamalinesari et al. (2015) researched written corrective feedback in an EFL context (English learners in Iran) and differentiate between direct (errors are corrected) and indirect (errors are pointed out and the students are expected to self-correct) feedback. The school context seems to be widely underresearched. Freedman (1987) is a well-known exception. Another aspect which attracts researchers’ interests is the perception of feedback. Saliu-Abdulahi (2019) focuses on how teachers and students perceive formative feedback practices in the EFL classroom. Furthermore, teachers were questioned on which feedback practices they employ in different didactic contexts. The research was conducted in Norway and concerns writing instruction in English in upper secondary school. Results show that written comments and endnotes (in Norwegian) were the most commonly used feedback forms. Teachers did not use peer feedback and oral conferencing regularly. Self-assessment was used more frequently. Teachers seem to be aware that feedback is beneficial, and some want to use more process-oriented approaches to writing instruction, including the use of portfolios and returning papers without grading them (53). However, it seems that their workload prevents them from integrating feedback more intensely in their teaching (55). Students highly appreciate feedback, but perceptions and preferences as far as feedback form is concerned vary. They confirmed that oral conferencing and peer feedback are not regularly used, but would help them to improve their writing if used regularly. They also voiced their desire to develop multiple drafts of a piece of writing and to receive feedback on the drafts before the grading process (54). In German-speaking countries, relevant research is comparatively rare. There is a growing amount of isolated research on written text feedback either in the school or teacher context (e.g. Esterl & Saxalber 2010; von Gunten 2015). Esterl and Saxalber (2010) investigated how teacher commentaries can shape pupils’ writing development and motivation to make revisions, while von Gunten (2015) tries to find out how the peer feedback strategies of teachers in training evolve over a period of three years. Knorr (2012) examined text commentaries by academics made in the context of the review process for the publication of an anthology. The Effect of Written Feedback Types on Students' Academic Texts 201 Busse (2013) touches on some aspects that are also addressed in the present pilot study. Hers is one of the few studies that deal with feedback at university level and with the question of how students perceive it. The results of her mixed-methods study focus on writers’ motivation. She investigated how German EFL students of English perceive feedback practices. Busse identifies four factors that resulted in a negative perception: students’ unfamiliarity with the grading system; insufficiently specific and thus unhelpful feedback; comprehensive rather than selective error correction; insufficient positive reinforcement (410). She highlights the need “for future investigations to take the interactive relationship between motivation, feedback, and the educational environment into account and outline pedagogical suggestions for how to gear feedback practices towards specific needs of first year students” (406). Additionally, there is growing research in the area of revision studies (Revisionsforschung) in the German-speaking context as well (see Höltmann 2012, Jantzen 2012a, Jantzen 2012b). There is very little research in German-speaking contexts on text feedback in higher education that takes the revision process into account, particularly regarding the question of how students tend to incorporate text feedback, either in native speaker contexts or in EFL contexts. Recently, von Gunten and Beyer (2018) have tried to summarise reasons for the lack of a working definition of feedback (“Textkommentierung”) within the German-speaking context. They argue that because feedback situations are extremely diverse, it is difficult to agree on what determines ‘good’ feedback. Therefore, measuring the effectiveness of feedback is equally complex. The only thing that can be agreed upon is that feedback has an effect and therefore an enormous potential within writing pedagogy (23). Although not suggesting a generally valid definition, von Gunten and Beyer list five aspects which they suggest should always be on the feedback givers’ minds, as they seem to influence any feedback situation: the purpose of the feedback (within the respective feedback situation), the relationship between feedback giver and feedback taker, the feedback giver’s attitude, the language phenomenon that triggers the feedback and how feedback is realised linguistically (13). Beyer (2018) has also developed a tool (InliAnTe) for analysing (peer) feedback instances within the context of a project aimed at developing feedback behaviour and feedback strategies of teachers in training. The tool consists of two grids, one of which focuses on the feedback trigger in the text and the other on how the feedback is linguistically realised. 3. Research Questions Written text commentary has not only become an essential instrument to help improve writing skills per se, but it also has a role in foreign lan- Günther Sigott, Melanie Fleischhacker, Stephanie Sihler, Jennifer Steiner 202 guage instruction. This is particularly important when it comes to languages which differ in terms of conventions of text organisation, as German and English do in terms of paragraph structure. Undergraduate students at university are novice writers who are unfamiliar with the conventions and norms of academic writing, but who need to adapt rapidly to this new context. They are often pressured by teachers and curricula to improve their writing skills as quickly as possible even though they receive limited feedback on their texts and have little time for revisions. Thus, the purpose of this study is to examine the effects of two types of written feedback provided by professional writing tutors on EFL students’ academic writing. In particular, we investigate the effect of directive and non-directive written feedback instances given by writing tutors on the revised versions of the students’ texts. Additionally, this study will open a tentative dialogue about students’ motivation for revision in a foreign language. The research questions are thus concerned with the effectiveness of the feedback in general and whether the effect depends on the type of feedback given as well as on the error that the feedback focuses on. In addition, the study addresses the question of whether the reasons for students ignoring the feedback might be related to error or feedback type. The study thus addresses the following research questions: RQ1: Does the feedback have any effect? RQ2: Does the effect of feedback depend on the kind of error that it focuses on? RQ3: Do the two feedback types (directive, non-directive) differ in their effect? RQ4: What are reasons for feedback takers not reacting to feedback? This pilot study is couched in a supportive, process-oriented approach that puts the dialogue between the feedback givers and the writers at its centre. By doing so, we hope to make the feedback process more transparent for feedback givers and feedback takers. A long-term goal is to develop pedagogical guidelines for feedback givers on the basis of the findings of a more substantial follow-up study. 4. Method 4.1. Participants Feedback Givers The three feedback givers are currently working at the writing centre of Klagenfurt University, Austria. They underwent a minimum of two years’ The Effect of Written Feedback Types on Students' Academic Texts 203 training as writing tutors based on contents such as writing didactics, writing research, linguistics, pedagogical approaches, and genre knowledge. They all have different specialisations but share the focus on writing in a foreign or second language. They had to undergo a practical training of at least 200 hours during this time, which mainly consisted of holding individual or group consultations with students, conceptualising and conducting workshops for students, and the participation in supervisions. In addition to their occupation at the writing centre, they continue to expand their knowledge of writing didactics and writing research through further education. Additionally, all of them are studying at the English Department of Klagenfurt University. Feedback Takers The study was conducted in the winter semester 2016/ 2017. At that time, the voluntary participants were studying English as a foreign language at the English Department of Klagenfurt University. Initially, the group consisted of nine undergraduates, but one student failed to provide their revised text version and was therefore excluded from the study. Six female and two male students remained. The students were between 22 and 47 years old. 4.2. Context and Materials The eight texts used in this study were written as a task for a Professional Writing Skills course. This mandatory class is part of the Bachelor’s degree programme English and American Studies and can typically not be taken any earlier than in the third semester as it has two other classes (Language Productive and Receptive Skills as well as Advanced Language Productive and Receptive Skills) as prerequisites. It is important to note that these three classes are part of a module aimed at familiarising students with academic writing standards as well as English text and language structures. The assignment for this particular course was to produce a small-scale academic paper of 800 words (+/ - 10%) serving as an exercise for future research-based writing projects. Students were supposed to generate a research question related to either the short story “The Ones Who Walk Away from Omelas” (1973) by Ursula K. Le Guin or one related to the concept of Carnism introduced by Melanie Joy in her article “From Carnivore to Carnist: Liberating the Language of Meat” (2001). Apart from their primary source, students had to include at least one secondary source and cite it according to the author-date system of the Chicago Manual of Style (16 th edition). Günther Sigott, Melanie Fleischhacker, Stephanie Sihler, Jennifer Steiner 204 4.3. Coding of the Data Since “[e]rror identification and error explanation with reference to the concept of norm […] plays an important role in formulating feedback to students,” (Sigott, Cesnik & Dobrić 2016: 79) the feedback instances in the eight texts were formulated following the Scope-Substance taxonomy for error identification. Scope “refers to the amount of context that is necessary in order for an error to become perceptible,” while substance “refers to the amount of text that needs to be changed so that the error will disappear” (ibid.: 80). Error identification always involves a combination of scope and substance. In our study, we focus on substance rather than scope, since this is what has to be changed by the feedback takers. All the errors that received feedback have repercussions beyond the sentence level and thus influence text cohesion and coherence. We decided to specifically focus on these error types, since it seems important to make writers aware of the effects of their errors on the entire fabric of the text. While for scope only text and paragraph were distinguished, four levels were distinguished for substance: text, paragraph, sentence, and word. This results in seven possible error types: text-text, text-paragraph, textsentence, text-word, paragraph-paragraph, paragraph-sentence, and paragraph-word. For purposes of analysis, the substance categories text and paragraph, as well as sentence and word are conflated to yield only two categories for the analysis. By doing this, we classify text and paragraph as higher-level errors while word and sentence are considered lower-level errors. The Feedback Type itself is classified as either non-directive or directive. We define a non-directive approach as identifying the error substance in the text without providing further suggestions for improvement, while the directive approach is defined as identifying the error substance in the text as well as providing suggestions for improvement. Feedback Effect is operationalised as either improvement or no improvement, and improvement is categorised as successful change while no improvement is subdivided into wrong change and unchanged. 4.4. Process For this pilot study, written text feedback, using the text commentary function in Microsoft Word, was provided on each students’ first draft by one of the three writing tutors one week prior to submission. Although peer feedback was encouraged by the writing instructor, no draft version that incorporates peer feedback was required before the students had to hand in their final versions. Non-directive and directive feedback types were used to approximately equal extents in this process. The eight revised text versions were then compared to the eight drafts. The written text commentary given to the students first identified the scope of the The Effect of Written Feedback Types on Students' Academic Texts 205 error type (text or paragraph) and then provided the writers with feedback on the substance level (word, sentence, paragraph, or text). These comments either did provide suggestions for improvement in terms of lexis, grammar, coherence and style (directive feedback) or did not provide any suggestions for improvement, thus being limited to identifying the part of the text that needed to be modified (non-directive feedback). The draft and revised versions of the texts were compared by all three writing tutors to arrive at a decision of improvement or nonimprovement. In addition, all three writing tutors examined each feedback instance to decide whether the feedback was directive or nondirective. In cases of initial disagreement, consensus was sought by negotiation. In rare cases in which this was not successful, a decision was made by majority vote. The same procedure was adopted to categorise errors as lower-level (sentence or word) or higher-level (text or paragraph) on the basis of their substance. Table 1 shows three examples in order to illustrate this process. Example 1 concerns the stylistic aspect of repetition, as pointed out by the tutor comment in the margin. This error has paragraph scope, since it does not affect the text as a whole, but rather the paragraph itself. As substance we identified word as the student merely needs to change a word in order to make the repetition less prominent. The feedback type is non-directive due to the fact that even though the repetition is pointed out by the tutor, no further suggestions for improvement are provided. As can be seen in the revised version, the error remained unchanged. Example 2 refers to the title of the student’s paper, which does not connect logically to the text that follows. Since this error affects the whole text, it has text scope. The substance is identified as sentence because reformulation of the title is required. In this case, the tutor comment is classified as directive because suggestions for improvement are given. Although the revised version shows a change in the title, it is still a wrong change as the error persists. The error in Example 3 revolves around the wrong use of a connector (linking word) between two ideas, which actually build on each other instead of one resulting from the other. The scope of this error is paragraph, since the error only becomes perceptible when one reads beyond the sentence level to paragraph level. The substance is categorised as word, because this is what has to be changed by the student. The tutor comment is classified as non-directive as the error is only indicated, but the suggestions for improvement are missing. As can be seen in the revised version, the feedback is successful, since the student used a more appropriate linking word instead. Günther Sigott, Melanie Fleischhacker, Stephanie Sihler, Jennifer Steiner 206 Ex. #1 #2 #3 Origin al Text P assag e The consumption of meat is also influenced by social constructs and the failing of the media has supported carnistic behaviours. Research has found that there are social structural influences for the consumption of animals. Firstly, the factor of social class appears to have a considerable influen ce on meat consumption. [emphasis added] Decisive Shapers in Aid of Carnism [title] In Islam, on the other side, the meat has to be Halal in order to be allowed to eat. Therefor e, also the Muslim belief forbids the consumption of pork … [emphasis added] Error Scope: paragraph Substance: word Scope: text Substance: sentence Scope: paragraph Substance: word Tutor Comment (in margin) (paragraph): repetition (text): This doesn't make sense in connection with the text that follows. Also, you could check up on collocations in order to improve clarity here. (paragraph): Incorrect use of linking word. F eedback Type non-directive directive non-directive Revis ed V ersion The consumption of meat is also influenced by social constructs and also the media has supported carnistic behaviours. Research has found that there are socially constructed influences for the consumption of animals. Firstly, the factor of social class appears to have a considerable influence on meat consumption. [emphasis add- Decisive Factors Working in Aid of Carnism In Islam, on the other hand, meat has to be Halal in order to be allowed to be eaten. A dditionally, the Muslim belief forbids the consumption of pork … [emphasis added] The Effect of Written Feedback Types on Students' Academic Texts 207 ed] Effect unchanged wrong change successful change Table 1: Examples of feedback instances and feedback effect 4.5. Methods of Data Analysis 1 In a first step, frequency statistics for the variable Feedback Effect were calculated and presented graphically. In a second step, Decision Tree Analysis as implemented in SPSS Version 24 was performed on the three variables Feedback Effect, Error Substance, and Feedback Type. In this design, Feedback Effect constitutes the dependent variable. The aim of the Decision Tree Analysis is to test whether groups of cases, i.e., feedback instances, differ significantly from each other in terms of their frequency distribution in the dependent variable. The purpose of the analysis is to determine which groups differ significantly from each other in terms of the ratio of feedback instances leading to successful change versus unchanged or wrong change. If two groups differ from each other with regard to this ratio, a node in the tree is formed. Groups whose ratios do not differ will not form a node. Differences in cell frequencies are tested for statistical significance by means of Chi-Square analysis with subsequent Bonferroni correction for multiple comparisons. In an entirely exploratory spirit, in order for even weak tendencies to become visible, the significance level was deliberately set to p = 0.6. Finally, reasons for failure to react to the feedback instances (Feedback Effect = ‘unchanged’) were investigated with regard to the variables Error Substance and Feedback Type in a separate analysis. Additionally, qualitative semi-structured trial interviews were carried out with some of the participant writers. In these short (up to ten minutes) interviews, the students were asked about one of the instances they did not react to in an attempt to begin to explore reasons for feedback rejection. Orthographic transcriptions, which only recorded longer breaks, overlaps in turns, and emotional responses such as laughter, were made. These were entered into Atlas.ti for a basic content analysis. The focus was on writers’ thought processes and motivation during the feedback and revision process. Only six out of eight students were asked about one particular feedback instance they did not respond to, since two of the writers had reacted to every feedback instance. 1 We would like to thank Dr. Hermann Cesnik, ZID, Klagenfurt University, for his invaluable advice on methodological aspects of this study. Günther Sigott, Melanie Fleischhacker, Stephanie Sihler, Jennifer Steiner 208 5. Results and Discussion To answer the four research questions, the relationships among the following three variables were examined: Feedback Effect (‘unchanged’, ‘wrong change’, ‘successful change’), Error Substance (‘sentence or word’, ‘text or paragraph’) and Feedback Type (‘directive’, ‘non-directive’). Figure 1: Frequencies for Feedback Effect As can be seen in Figure 1, most feedback instances (53.33%) have led to successful changes in the student texts. Considerably fewer feedback instances, namely 28.89 percent, did bring about changes in the text, but these either still contained an error or introduced a new error. Feedback instances leading to no change at all are by far the least frequent with 17.78 percent. This is clear evidence of a strong positive effect of feedback in general and suggests a strong positive answer to RQ1. The Effect of Written Feedback Types on Students' Academic Texts 209 Figure 2: Decision Tree for Feedback Effect (recoded) Figure 2 shows the result of the exploratory Decision Tree Analysis. For the purposes of this analysis, no changes (unchanged) and wrong changes were conflated by recoding the original variable Feedback Effect into a new binary variable Effect_rec with the two categories ‘unchanged or wrong change’ and ‘successful change’. Similarly, the original variable Error Substance was recoded into a new binary variable Substance_rec with the two categories ‘sentence or word’ and ‘text or paragraph’. Since both independent variables are binary after recoding, the feedback instances fall into one of four groups: ‘sentence or word’ - ‘non-directive’, ‘sentence or word’ - ‘directive’, ‘text or paragraph’ - ‘non-directive’, ‘text or paragraph’ - ‘directive’. Figure 2 shows that 46.7 percent of feedback instances led to no improvement (‘unchanged or wrong change’) while 53.3 percent led to improvement (‘successful change’). If the feedback instances are divided into two groups according to the error they focus on, namely low-level (‘sentence or word’) or high-level (‘text or paragraph’), the ratio of improvement to no improvement is markedly different in the two groups. In the low-level group, the percentage of feedback instances leading to improvement (‘successful change’) is 60.7, while the Günther Sigott, Melanie Fleischhacker, Stephanie Sihler, Jennifer Steiner 210 percentage of feedback instances that led to no improvement (‘unchanged or wrong change’) is only 39.3. This is different for the high-level group. There, the percentage of feedback instances leading to improvement is lower (37.9%) than that leading to no improvement (62.1%). Obviously, feedback is more effective for low-level errors than for high-level errors. Therefore, the effect of feedback is determined significantly (Adj. P-value = 0.043) by Error Substance. This suggests a clear positive answer to RQ2. At the second level of the tree diagram, an effect of Feedback Type (‘directive’ vs ‘non-directive’) becomes visible as well. In the low-level group, non-directive feedback leads to considerably more improvement (69.7%) than to no improvement (30.3%), while directive feedback brings about as much improvement as non-improvement (50%). By contrast, in the high-level group of feedback instances, both feedback types lead to more non-improvement than to improvement. This is more pronounced for directive feedback, where 66.7 percent of feedback instances lead to no improvement while only 33.3 percent lead to improvement. Altogether this effect is weaker than that of Error Substance. It is stronger in the low-level group (Adj. P-value = 0.117) than in the high-level group (Adj. P-value = 0.514). However, it needs to be pointed out that the error probability for both nodes are p = 0.117 and p = 0.514 respectively, indicating that there may not be a meaningful difference at all. Altogether, this indicates that after Error Substance, Feedback Type may also influence Feedback Effect. Non-directive feedback seems to be more effective than directive feedback. This is more pronounced at low Error Substance levels than at high Error Substance levels. This suggests a tentative positive answer to RQ3. In a different Decision Tree Analysis, neither Error Substance nor Feedback Type were found to have a significant effect on feedback takers’ failure to react to feedback at all. Thus, the reasons for this have to be sought for elsewhere. This provides only a partial answer to RQ4, but clearly shows a need for more research of a qualitative type to shed further light on this issue. As a tentative first step, six students were questioned in trial interviews about one of the instances they had not reacted to in any way. From those interviewed, only two overtly disagreed with the feedback given to them, while only one of them still rejected the commentary after discussing it with the feedback giver. In one case, the rejection resulted from a failure to reconcile the feedback with the input on linking words and phrases from the writing class. Two students had simply failed to see the problem, namely repetition of words and missing information, in their respective texts. Highlighting the lexical items in question, so one of the students, would have made the repetition more obvious in his case. In another text, the feedback had not been given in response to the first occurrence of the problem, which led to confusion for the feedback taker. Due to the amount of feedback that had been giv- The Effect of Written Feedback Types on Students' Academic Texts 211 en for the purpose of this study, one student had simply overlooked the particular feedback instance in question. Most of the students who had been interviewed suggested that they would have integrated even more of the feedback if they had received a more thorough explanation with regard to the problem(s) in their texts. It seems that there may be a type of error where directive feedback might be more helpful after all. 6. Conclusion Overall, then, feedback is effective, but more so at word and sentence level than at paragraph and text level. In addition, non-directive feedback is more effective than directive feedback, more so at word and sentence level than at text and paragraph level. Reasons for non-responses, however, are not associated with Feedback Type or Error Substance. Some reasons for feedback having more effect at word and sentence level could be related to students’ language proficiency, knowledge about discourse structure, and personal agency. Lower level errors are easier to correct and do not require changing the structure or the argument of the entire text. Also, undergraduate students are trained to look at lowerorder concerns rather than higher-order concerns since this is often the focus in schools (Grieshammer, Liebetanz, & Peters 2012). By contrast, paragraph and text level present a greater and relatively new challenge for EFL writers of English at university level because German discourse structures deviate considerably from English ones. Such problems, however, might be resolved as future students will have acquired knowledge about Anglophone discourse structures in order to pass the recently introduced Austrian standardised school-leaving exam (BMBWF, n.d.). Even though it might seem surprising that non-directive feedback is more effective, it is conceivable that pre-formulated suggestions made by the tutors in their own writing style tend to be rejected by the student writers. This might be due to a wish for personal agency (Reid 1994, Straub 1997), and the development of an individual voice (see Elbow 1973, Tardy 2006, Girgensohn 2007, Mertlitsch 2013). According to Elbow (1973), the individual voice of a writer is created as a result of an interaction between the author and what has been said. Rather than simply implementing something ready-made, students prefer to choose freely when and how they take on suggestions. The concept of appropriation, appropriating another person’s words for one’s own argument, is also closely linked to questions of voice and power. Power relations are often involved in feedback situations as teachers have more authority and students might feel pressured to take on their suggestions (Tardy 2006: 61-63). This is not the case in a peer-tutor context, however, where feedback givers never take over ownership of the text. Also, according to Vygotsky’s (1978) principle of scaffolding, learners will take on guidance Günther Sigott, Melanie Fleischhacker, Stephanie Sihler, Jennifer Steiner 212 as long as they need it but will resort to personal choices as soon as they are able to manage on their own. Again, this tendency is accentuated at lower levels, and less prominent at text and paragraph levels. Writers, it seems, feel more confident in tackling sentence and word problems, but are still unfamiliar with English academic discourse structures. Generally, text development problems are much more difficult to resolve for inexperienced writers. With the limited data available, there is no visible association between non-responses and either Feedback Type or Error Substance. The reasons, however, could be the subject of a future study involving qualitative introspective methods such as stimulated recall and revise-and-resubmit letters especially designed for the purpose of investigating non-responses to feedback instances. In a replication of the study, it might be worthwhile to provide feedback for lower-level and higher-level substance in two separate phases or by using a colour-coding system in order to make the revision process less strenuous for the novice writers. In the future, the results of this pilot study, and a more extensive follow-up study, which is currently in progress, might help feedback takers to improve their revision skills and feedback givers (writing instructors, writing tutors, and teachers in general) to refine and expand their skill sets. Feedback and writing processes could be made more transparent and successful by developing in feedback givers a heightened awareness of the effects feedback (both in its form and content) can have on writers’ texts and editing behaviour. Feedback givers should benefit from knowing which areas tend to be more resistant to feedback and therefore need particular attention. The preliminary results already point to a need for further research with a higher number of feedback instances in order to determine why feedback at higher substance levels is less effective and why non-directive feedback tends to be more effective. Additionally, it would be interesting to determine the role that error scope plays in the effectiveness of feedback or whether the effectiveness of non-directive feedback is related to the question of personal agency. References Berggren, Jessica (2013). Learning from giving feedback: Insights from EFL writing classrooms in a Swedish lower secondary school. Sweden: Stockholm University. Beyer, Anke (2018). “InliAnTe: Instrument für die linguistische Analyse von Textkommentierungen”. Linguistik online 4.18: 15-40. [online] http: / / dx.doi.org/ 10.13092/ lo.91.4394 [2019, April 4] BMBWF. (n.d.). SRDP. Standardisierte Reife- und Diplomprüfung. [online] https: / / www.srdp.at/ [2019, April 4] Brannon Lil, & C.H. Knoblauch (1982). “On students’ rights to their own texts: A model of teacher response”. College Composition and Communication 33.2: 157- 166. The Effect of Written Feedback Types on Students' Academic Texts 213 Brown, Dan (2012). “The corrective feedback debate: Next steps for classroom teachers and practitioners”. TESOL Quarterly 46.4: 861-867. Busse, Vera (2013). “How do students of German perceive feedback practices at university? A motivational exploration”. Journal of Second Language Writing 22: 406-424. Connor, Ulla & Karen Asenavage (1994). “Peer response Groups in ESL writing classes: How much impact on revision? ” Journal of Second Language Writing 3: 257-176. Connors, Robert J. & Andrea A. Lunsford (1993). “Teachers’ Rhetorical Comments on Student Papers”. College Composition and Communication 44.2: 200-223. Conrad, Susan & Louis M. Goldstein (1999). “ESL Student Revision after Teacher- Written Comments: Texts, Contexts, and Individuals”. Journal of Second Language Writing 8.2: 147-179. Elbow, Peter (1973). Writing without Teachers. New York: Oxford University Press. Esterl, Ursula & Annemarie Saxalber (2010). “‘Inhaltlich hast du sehr gut gearbeitet...’: Funktionen und Qualität eines förderorientierten LehrerInnenkommentars”. In: Schreibprozesse begleiten: Vom schulischen zum universitären Schreiben, 181-214. Eds. Ursula Esterl & Annemarie Saxalber. Innsbruck/ Wien/ Bozen: StudienVerlag. Ferris, Dana R. (1997). “The influence of teacher commentary on student writing”. TESOL Quarterly 31.2: 315-339. Ferris, Dana R. (2014). “Responding to student writing: Teachers’ philosophies and practices”. Assessing Writing 19: 6-23. Ferris, Dana R., Jeffrey Brown, Hsiang Liu, Maria Eugenia & Arnaudo Stine (2011). “Responding to L3 students in college writing classes: Teacher perspectives”. TESOL Quarterly 45.2: 207-234. Fitzgerald, Jill (1987). “Research on revision in writing”. Review of Educational Research 57.4: 481-506. Freedman, Sarah W. (1987). Response to Student Writing. Urbana, IL: NCTE. Girgensohn, Katrin (2007). Neue Wege zur Schlüsselqualifikation Schreiben. Autonome Schreibgruppen an der Hochschule. Wiesbaden: VS Research. Grieshammer, Ella, Franziska Liebetanz & Nora Peters (2012). Zukunftsmodell Schreibberatung: Eine Anleitung zur Begleitung von Schreibenden im Studium. Hohengehren: Schneider Verlag. Harran, Marcelle (2011). “What higher education students do with teacher feedback: Feedback-practice implications”. Southern African Linguistics and Applied Language Studies 29.4: 419-434. Höltmann, Melina (2012). “Lernchancen durch das Überarbeiten mit der Methode ‘Tipps am Rande’ - eine empirische Untersuchung”. In: Überarbeiten lernen - Überarbeiten als Lernen, 179-203. Eds. Petra Hüttis-Graff & Christoph Jantzen. Stuttgart: Fillibach bei Klett. Hyland, Fiona & Ken Hyland (2001). “Sugaring the pill: Praise and criticism in written feedback”. Journal of Second Language Writing 10: 185-212. Jamalinesari, Ali, Farahnaz Rahimi, Hsbib Gowhary & Akbar Azizifar (2015). “The effect of teacher-written direct vs. indirect feedback on students’ writing”. Procedia - Social and Behavioral Sciences 192: 116-123. Jantzen, Christoph (2012a). “Was Überarbeitungskompetenz ist”. In: Überarbeiten lernen - Überarbeiten als Lernen, 343-346. Eds. Petra Hüttis-Graff & Christoph Jantzen. Stuttgart: Fillibach bei Klett. Jantzen, Christoph (2012b). “Woher wir meinen zu wissen wie Lernende ihre Texte überarbeiten: Zur Methodologie der Datenerhebung in einer didaktisch Günther Sigott, Melanie Fleischhacker, Stephanie Sihler, Jennifer Steiner 214 ausgerichteten Überarbeitungsforschung”. In: Überarbeiten lernen - Überarbeiten als Lernen, 11-36. Eds. Petra Hüttis-Graff & Christoph Jantzen. Stuttgart: Fillibach bei Klett. Knorr, Dagmar (2012). “Textkommentierungen: Formen und Funktionen”. In: Schreiben unter Bedingungen von Mehrsprachigkeit, 75-98. Eds. Dagmar Knorr & Annette Verhein-Jarren. Frankfurt am Main, Berlin: Peter Lang. Kwangsu, Cho, Christian D. Schunn, Roy W. Wilson (2006). “Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives”. Journal of Educational Psychology 4: 891-901. Leki, Ilona (1990). “Potential problems with peer responding in ESL writing classes”. CATESOL Journal 3: 5-19. Leki, Ilona (1991). “The preferences of ESL students for error correction in college-level writing classes”. Foreign Language Annals 24: 203-218. Liu, Jun & Jette G. Hansen (2002). Peer Response in Second Language Writing Classrooms. Michigan: University of Michigan Press. Mertlitsch, Carmen (2013). “Die eigene Stimme finden”. Ide Zeitschrift für den Deutschunterricht in Wissenschaft und Schule 3: 80-87. Nelson, Melissa M., & Christian D. Shunn (2009). “The nature of feedback: how different types of peer feedback affect writing performance”. Instructional Science 37.4: 375-401. Patthey-Chavez, Genevieve & Dana R. Ferris (1997). “Writing conferences and the weaving of multi-voiced texts in college composition”. Research in the Teaching of English 31.1: 51-90. Reid, Joy (1994). “Responding to ESL student’s texts: The myths of appropriation”. TESOL Quarterly 28.2: 273-292. Saliu-Abdulahi, Drita. (2019). Teacher and Student Perceptions of Current Feedback Practices in English Writing Instruction. Norway: University of Oslo. Sigott, Günther, Hermann Cesnik & Nikola Dobrić (2016). “Refining the Scope- Substance Error Taxonomy: A closer look at substance”. In: Corpora in Applied Linguistics - Current Approaches, 79-94. Eds. Nikola Dobrić, Eva-Maria Graf & Alexander Onysko. Newcastle upon Tyne: Cambridge Scholars Publishing. Sommers, Nancy (1982). “Responding to student writing”. College Composition and Communication 33.2: 148-156. Straub, Richard (1996). “The concept of Control in teacher response: Defining the varieties of ‘directive’ and ‘facilitative’ commentary”. College Composition and Communication 47.2: 223-251. Straub, Richard (1997). “Students’ reactions to teacher comments: An exploratory study”. Research in the Teaching of English 31.1: 91-119. Tardy, Christine (2006). “Appropriation, ownership, and agency: Negotiating teacher feedback in academic settings”. In: Feedback in Second Language Writing: Contexts and Issues, 60-79. Eds. Ken Hyland & Fiona Hyland. Cambridge: Cambridge University Press. Tschudi, Susan, Heidi Estrem & Patti-A. Hanlon (1997). “Unsettling drafts: Helping students see new possibilities in their writing”. The English Journal 86.6: 27-33. von Gunten, Anne. (2015). “Wie und wozu angehende Lehrpersonen Texte kommentieren: Forschungsdesign und Analyse-Instrumente eines Projektes”. Zeitschrift Schreiben: Schreiben in Schule, Hochschule und Beruf 1: 1-7. von Gunten, Anne & Anke Beyer (2018). “Schriftliche Textkommentierungen. Versuch einer Begriffsklärung aus schreibdidaktischer Sicht”. JoSch - Journal der Schreibberatung 16.18: 11-26. The Effect of Written Feedback Types on Students' Academic Texts 215 Vygotsky, Lev S. (1978). Mind in Society: The Development of Higher Psychological Processes. Cambridge, MA: Harvard University Press. Young Huan, Cho & Cho Kwangsu (2011). “Peer reviewers learn from giving comments”. Instructional Science 39: 629-643. Zamel, Vivian (1985). “Responding to student writing”. TESOL Quaterly 19.1: 79- 101. Texts used as prompts for class assignment Le Guin, Ursula K. (2004). “The Ones Who Walk Away from Omelas”. In: The Contemporary American Short Story, 314-320. Eds. Bich Minh Nguyen & Porter Shrev. New York: Pearson Education. Joy, Melanie (2001). “From Carnivore to Carnist: Liberating the Language of Meat”. [online] http: / / www.satyamag.com/ sept01/ joy.html [2019, January 20] Günther Sigott Melanie Fleischhacker Stephanie Sihler Jennifer Steiner Department of English University of Klagenfurt Austria