Reflections on the Slovenian Dictionary User Study of 2018: What does it tell us? Alenka Vrbinc, Donna M. Farina and Marjeta Vrbinc This paper reports on a qualitative study of dictionary use at the School of Economics and Business, University of Ljubljana, Slovenia, during which nine students were given look-up tasks with the online Merriam-Webster Learner’s Dictionary. The study employed a combination of research methods: semistructured oral interviews and the researchers’ direct observation of the participants as they looked up words. As the students completed these tasks, they were observed and questioned about their habits of dictionary use, their lookup experience, and their perceptions of the utility and quality of the dictionary definitions and examples. The results provide insight into the efficacy of the specific dictionary used. In addition, the study reveals much about how these students regard dictionaries and how they approach their use. Many of the participants had no relationship with dictionaries and no real understanding of their purpose. Their comments demonstrate that they are “demanding” users with very firm ideas and high expectations about the type of information they wish to receive in an online dictionary - and how they prefer to have it delivered. Some recommendations are made for those involved in learner lexicography concerning the improvement of part-of-speech information to make lookup easier, improvement of dictionary examples and improvement of the way dictionary information is presented. This paper also discusses what the takeaways are for concerned dictionary makers; in particular, it will reflect on how students should be taught about dictionaries today - if we still want them to use dictionaries tomorrow. 1. Introduction Within the framework of a joint project between Slovenia and the United States (see Acknowledgements), a qualitative dictionary study was carried out in March 2018 with nine students in the School of Economics and Business at the University of Ljubljana. The students majored in different areas AAA - Arbeiten aus Anglistik und Amerikanistik Agenda: Advancing Anglophone Studies Band 49 · Heft 1 Gunter Narr Verlag Tübingen DOI 10.24053/ AAA-2024-0004 Alenka Vrbinc, Donna M. Farina and Marjeta Vrbinc 86 of business and economics and were not specialists in English. The dictionary used in the study was the Merriam-Webster Advanced Learner’s Dictionary, the only American learner’s dictionary; the study used the online variant called the Merriam-Webster Learner’s Dictionary (MWLD; https: / / learn ersdictionary.com/ ). 1 The goal of this study was to learn about the students’ habits, impressions, and look-up challenges while using an online dictionary. Before turning to the study itself, it will be helpful to provide first some background information on learner lexicography. What follows is a brief history of learner’s dictionaries as well as a review of research studies that have investigated dictionary users. 2. The history of English learner’s dictionaries Learner’s dictionaries can be traced back to the 1920s and 30s and resulted from the experience of British educators in English language teaching in Japan. The pioneers in learner lexicography were Michael West, Harold E. Palmer, and Albert Sydney Hornby, who are considered today as the founding fathers of applied linguistics. They conducted research - vocabulary studies - in Japan (Palmer and Hornby) and India (West), to determine the effects of using a controlled (i.e., limited and simpler) vocabulary in dictionaries. The earliest learner’s dictionaries were developed in the 1930s from the vocabulary studies of West, Palmer, and Hornby, as well as the U.S. linguist Edward L. Thorndike (cf. Battenburg 1994; Cowie 1998; Landau 2001). Michael West concentrated his research on how existing reading texts for learners of English could be improved; he is considered ahead of his time in his ideas about facilitating the development of foreign-language reading (Battenburg 1994). The first principle he proposed was the simplification of vocabulary by removing rare or dated words and using more common or more modern equivalents instead. This formed the basis for what is now known as a controlled defining vocabulary, a list of 2,000- 3,000 high-frequency words that the learner is expected to know sufficiently well to be able to understand any definition in a learner’s dictionary. West’s second principle was to distribute any new words (beyond the controlled vocabulary) appearing in a reader in such a way that they do not appear too close together; in other words, he introduced new words more slowly and distributed them throughout a text (Battenburg 1994). This allows the user of the text to absorb and practice them thoroughly. In 1 As of June 2023, this link still leads to the dictionary of the original research study. However, the dictionary has been rebranded as “The Britannica Dictionary” (https: / / www.britannica.com/ dictionary) by Encyclopædia Britannica, Inc., the sister company of Merriam-Webster. A Merriam lexicographer confirmed that the dictionary itself has not changed. Reflections on the Slovenian Dictionary User Study of 2018 87 the English readers prepared by West, the number of new words decreased at the same time that the total number of words in the running text increased significantly. On the basis of his research into the benefits of vocabulary control (i.e. limitation), West compiled The New Method English Dictionary (1935) in collaboration with James Gareth Endicott. In this - the very first - learner’s dictionary, West introduced the simplified defining vocabulary outlined in his Definition Vocabulary, published in the same year (Rundell 1998; Cowie 1999; Atkins & Rundell 2008; Svensén 2009). Definition Vocabulary is a research report describing how the vocabulary was selected, checked, and revised. It originally consisted of 1,799 words but was later reduced to 1,490 words used to define the 23,898 vocabulary items included in The New Method English Dictionary. Today, all learner’s dictionaries make use of a controlled defining vocabulary. In order to understand how a controlled vocabulary drastically changes the way dictionary definitions look, we repeat below a timely example also noted by Battenburg (1994) that compares the definition for the word vaccinate in West’s New Method Dictionary with that in a dictionary intended for native English speakers (Webster’s New International Dictionary): New Method Dictionary (West & Endicott 1935) vaccinate ... to protect against smallpox (= a dangerous disease causing spots on the skin) by putting into the arm a liquid obtained from cows which have had the disease. Webster’s New International Dictionary, 2 nd edition (Nelson 1934) vaccinate ... to inoculate with cowpox vaccine, esp. in order to prevent or mitigate any attack of smallpox; hence, to inoculate with any vaccine, or, loosely, any virus, esp. as a preventive measure. Harold Palmer was also interested in vocabulary control. Battenburg (1994) notes that Palmer and West “considered the experienced classroom teacher’s ‘vocabulary sense’ to be as important as the relative frequency of vocabulary items; ” the practical classroom experience of both men left an imprint on their research and theories. In 1924, Palmer published A Grammar of Spoken English, a scholarly work treating sentence patterns in a series of tables. This laid the foundation for the verb patterns presented in early learner’s dictionaries (see the discussion of Hornby, below). In 1938, Palmer published A Grammar of English Words, a highly innovative learner’s dictionary (despite the title suggesting a grammar book). This is a pioneering encoding dictionary (i.e., a dictionary that enables users to create or encode their own texts). In it, Palmer made use of his own structured word list (Thousand-Word English, 1937), which he prepared with A.S. Hornby. Palmer’s A Grammar of English Words is the first dictionary ever to label countable nouns (such as book/ books) and uncountable nouns (such as Alenka Vrbinc, Donna M. Farina and Marjeta Vrbinc 88 rice). It also employed an original approach to indicating variation in examples. The Oxford Advanced Learner’s Dictionary of 1974 provides good illustrations of how variation in examples operates in learner’s dictionaries. For the noun collapse, this dictionary has: “the collapse of a table/ tent/ tower …; the collapse of their plans/ hopes.” For the verb refresh, this dictionary gives: “refresh oneself with a cup of tea/ a warm bath.” Variation in examples with the adjective lofty is given as “a lofty tower/ mountain.” Palmer’s approach to examples left an indelible mark on Hornby’s later Idiomatic and Syntactic English Dictionary (1942), the precursor to the modern-day Oxford Advanced Learner’s Dictionary. A. S. Hornby played a leading role in phraseological research and also engaged in research on word lists (Cowie 1999). He was the originator of the 1,000-word vocabulary which impacted Palmer’s A Grammar of English Words (1938). In 1937, together with E. V. Gatenby and H. Wakefield, Hornby began work on a new type of general monolingual dictionary aimed at foreign learners. It was published as The Idiomatic and Syntactic English Dictionary (1942) and later reissued by Oxford University Press as A Learner’s Dictionary of Current English (1948), the first edition of what would become the well-known Oxford Advanced Learner’s Dictionary (OALD). Hornby’s dictionary is renowned for building on the work of Palmer and including a full-fledged verb-pattern scheme (Cowie 1998, 1999). We can turn to a later version of Hornby’s original dictionary, the OALD of 1974, for examples of the type of information about verb patterns that became standard in learner’s dictionaries. The 1974 dictionary’s front matter (pp. xvi-xvii) lists, explains, and exemplifies as many as 51 different verb patterns. The verb pattern “VP6E” is explained as “S + NEED/ WANT, etc + gerund, etc (passive),” which means that after need/ want and won’t/ wouldn’t bear, the gerund is equivalent to a passive infinitive. This pattern is exemplified in the dictionary by: ‣ My shoes want mending. ‣ He’ll need looking after. ‣ His language wouldn’t bear repeating (= His language wouldn’t bear (= was too bad) to be repeated.) All the editions of the OALD published over the years have been great commercial successes in the field of English language teaching; the tenth edition was published in 2020. A. S. Hornby achieved a unique status and authority in learner lexicography; to this day, his name is synonymous with learner’s dictionaries (Landau 2001). The period of the 1960s and 70s is considered the second generation of learner’s dictionaries. In 1978, the OALD was challenged by a competitor when the Longman Dictionary of Contemporary English was published. Well Reflections on the Slovenian Dictionary User Study of 2018 89 into the Information Age, the 1980s mark a watershed in learner lexicography. In 1987, the first entirely corpus-based dictionary appeared on the market, the Collins COBUILD English Language Dictionary. A corpus is a database, a collection of thousands (or more) of texts that allow a dictionary maker to look up words and then examine word usage in context. The Collins company calls the 1987 dictionary “the first of a new generation of dictionaries that were based on real examples of English rather than on compilers’ intuition” (Collins English Language Teaching Blog (online)). COBUILD was not only innovative in that it used a corpus, but it also used a new style of definitions called full-sentence definitions. Full-sentence definitions were developed on the basis of the findings of spoken discourse analysis and are characterized by the use of complete sentences that include the definiendum (i.e., the word that is being defined). The full-sentence definition resembles ordinary speech and consists of two parts: the first part, in which the definiendum is embedded, exemplifies usage and is called the contextualization part, while the second part supplies the actual definition and is called the explanatory part (Atkins & Rundell 2008: 441; Svensén 2009: 235, 236). At the same time, the fourth edition of the OALD was published (1989) and the second edition of Longman appeared in print (1987). These dictionaries of the 1980s are considered the third generation of learner’s dictionaries. The end of the twentieth century and beginning of our new century brought more competition with other publishers entering the scene. Cambridge University Press entered the market in 1995 with the Cambridge International Dictionary of English, renamed the Cambridge Advanced Learner’s Dictionary for the 2005 (second) edition. The latest newcomers to the field of learner lexicography are the Macmillan English Dictionary, first published in 2002, and the first American contribution, the Merriam-Webster Advanced Learner’s English Dictionary, which first appeared in 2008. The most recent edition of Longman is the seventh edition published as an e-book in 2023, that of Collins COBUILD is the tenth edition published in 2023, and the latest from Cambridge University Press is the fourth edition (2013). Merriam-Webster’s second edition of the Advanced Learner’s English Dictionary appeared in print in 2016. The end of the 20th century did not only see competition among English learner’s dictionaries but also the first transformation of medium, i.e., paper dictionaries were turned into CD- ROMs with technical challenges and new potentials. All six of these learner’s dictionaries are now available online. 2 Five of these online dic- 2 The online versions of these dictionaries are available as follows: http: / / www.oxfordlearnersdictionaries.com/ definition/ english/ ; http: / / www.ldoceonline.com/ ; http: / / www.collinsdictionary.com/ dictionary/ english; http: / / dictionary.cambridge.org/ dictionary/ british/ ; http: / / www.macmillandictionary.com/ ; and https: / / www.britannica.com/ dictionary. Alenka Vrbinc, Donna M. Farina and Marjeta Vrbinc 90 tionaries include residue from the above-mentioned technical transformation processes, which take time to complete and are costly. What is more, because the older lexicographers do not live in the same media space as the much younger dictionary users, the decisions about how to configure new online formats are not always optimal for modern users. It is uncertain how many more print editions from the “Big Six” publishers we can expect in the future. Already twelve years ago, the Macmillan company announced in its blog “the end of the printed dictionary” (Rundell 2012). Hornby, Palmer, and West, with their deep understanding of the problems faced by English learners, undertook the meticulous study of vocabulary, verb patterns, variation in common word combinations, and many other areas of language. Their painstaking work laid the foundation for contemporary learner lexicography. Today, when the students at the University of Ljubljana or elsewhere in the world type a word in the search box of online English learner’s dictionaries, they are certainly not aware of how many layers of history they are accessing. 3. Studies of dictionary use Before the 1960s, learner lexicography relied on empirical but anecdotal observations by dictionary makers. More systematic research on dictionary users and on the dictionary consultation process began in the 1960s, concurrent with the advent of the second generation of learner’s dictionaries (see above). An early user study of monolingual English speakers was carried out by the American lexicographer Clarence Barnhart, to find out what dictionary users look up most commonly. Barnhart circulated a written questionnaire in “99 colleges in 27 states” (1962: 162) in the United States, and concluded that American college freshmen most often use dictionaries for meaning and spelling. In the 1970s, research began to focus on who uses dictionaries, what for, and how. Tomaszczyk (1979) was the first to study the needs of foreign learners of English; he utilized a written questionnaire and identified translation, writing, and reading as the most common reasons that they used the dictionary. Studies of dictionary users, especially of the users of learner’s dictionaries, proliferated in the 1980s and 1990s and have continued to appear frequently up to the present. In the Internet era, the main intent of research into dictionary use is to find out in which situations dictionaries are consulted and how they are used; investigators also examine different components of the online dictionary environment to determine how successfully users are accessing each component (Müller-Spitzer, Koplenig & Wolfer 2018). Similarly, Kosem et al. (2019) investigated how often, on which devices, and in which situations monolingual dictionaries were used. Most studies are of English dictionaries (monolingual, bilingual, bilingualized) (Béjoint 2010) and of their users. Béjoint (1994) discussed studies showing Reflections on the Slovenian Dictionary User Study of 2018 91 that some types of dictionary information are consistently misunderstood and some types are rarely used; he emphasized the complexity of the dictionary consultation process and maintained that there is a disconnect between the reference skills of the dictionary user and the expectations of the dictionary maker. A seminar sponsored by the European Association for Lexicography addressed research into dictionary use and design (Cowie 1987); Atkins (1998) honed in on the use of dictionaries by translators and language learners. Dolezal and McCreary’s (1999) critical bibliography of pedagogical lexicography discussed numerous studies. Béjoint (2010) presented a detailed review of topics addressed by different dictionary researchers. There are studies of reference needs (what do the users look up; are dictionaries used for meaning and spelling only), studies of how dictionaries are used (the look-up process; general look-up strategies; understanding the definitions; where do the users look for multiword items), and studies of how dictionaries help (is there a better definition style; how should information for expression be conveyed; does the dictionary help users understand words; dictionaries and understanding a text). Below, we mention briefly some well-known recent dictionary user studies and compare them with the present study, of Slovenian students in economics and business. The user study under focus here, the first of its kind to be carried out in Slovenia, examined the dictionary consultation process. The researchers observed and took notes as nine Slovenian advanced learners of English searched for specific, infrequent meanings in an online learner’s dictionary, the Merriam-Webster Learner’s Dictionary (MWLD). Following the look-up process, we asked student-participants a series of questions. Unlike the studies of Lew (2011) or Lorentzen and Theilgaard (2012), the present study did not address how users find or access dictionaries. Rather, in this study, the dictionary was immediately presented to them on-screen; they were not given the choice to use a print dictionary instead or an online dictionary from another publisher. The present study also contrasts with studies that compare two or more different dictionaries. For example, Herbst (1996) looked at collocations (i.e., words that appear frequently together, such as narrow margin or considerable margin 3 ), definitions, policies related to examples, and other matters in four learner’s dictionaries; de Schryver and Prinsloo (2011) investigated the definitions from three different Dutch dictionaries. Liu, Zheng and Chen (2019) sought to discover how Chinese EFL learners use and view smartphone dictionaries. In contrast with Gouws (2014) and Lew and de Schryver (2014), the Slovenian study did not have as one of its goals the examination of the design 3 The online Collins COBUILD dictionary has a feature called “Related Word Partners” to illustrate collocations (i.e., combinations) of words. For the word margin, they list 19 different ones (https: / / www.collinsdictionary.com/ us/ dictionary/ english/ margin). Alenka Vrbinc, Donna M. Farina and Marjeta Vrbinc 92 or format of online dictionary information; nevertheless, our results show that design is a crucial factor for the users we worked with. There are numerous studies that have explored the ordering of meanings or salience of meaning positions (see Dziemianko 2014 for a summary of such studies); the users in the present study provided new findings about the significance of the order in which information is presented in the online dictionary environment. Our study participants were not shy in sharing their suggestions for improving online learner’s dictionaries; they made clear the diverse types of expectations they brought with them to the dictionary. 4. Methodology of the Slovenian study 4 As indicated in the Introduction, this study was carried out in 2018 under a joint project between Slovenia and the United States. The participants were nine students in the School of Business and Economics at the University of Ljubljana. While these students used English daily, they were by no means specialists in the language. The researchers wished to discover what the participants expect from a dictionary. The researchers observed the participants’ dictionary look-up process and used prepared questions to encourage participants to recommend improvements in online learner’s dictionaries. The online Merriam-Webster Learner’s Dictionary (MWLD) used in the study was selected because it is unknown to most university students in Slovenia. In fact, none of the study participants had familiarity with it. The study employed a combination of research methods: semi-structured oral interviews (Qu & Dumay 2011) and the researchers’ direct observation of the participants as they looked up words (Hatherall 1984). During the semi-structured interviews, the participants were asked questions that corresponded to four pre-identified topics (see below); such questions received follow-ups when necessary, depending on the context of a given moment (Hannabuss 1996). While they were looking up the words in the dictionary, the students were encouraged to think aloud (Wingate 2002) and explain what they were doing at any given time. The four pre-identified topics were: 1. Habits of Dictionary Use 2. Look-up Ability of Participants 3. Perceptions of Usefulness and Quality of Definitions 4. Perceptions of Usefulness and Quality of Dictionary Examples 4 For more detailed information on the methodology and results of this study, see Farina, Vrbinc and Vrbinc (2019), Vrbinc, Farina and Vrbinc (2022), Vrbinc, Vrbinc and Farina (2023). Reflections on the Slovenian Dictionary User Study of 2018 93 There were 14 interview questions in all. The first six were general, designed to obtain background information about the students’ previous use of dictionaries. They covered dictionary use habits, participants’ satisfaction with dictionaries they use, and speed/ efficiency of look-up. One question asked what students dislike in the dictionaries they use (Topic 1: Habits of Dictionary Use). After answering these general questions, each participant proceeded to read a series of very short texts with targeted words in them. In all, there were nine common words from standard English for the students to look up. However, in the nine short texts that students were given, these words were not used in their common meanings, but an infrequent meaning was used in each text. For example, the noun plug most commonly means something that blocks up a hole, or an electrical device inserted into a wall outlet. In this study, the students were given a reading text with plug in the infrequent meaning: a praising advertisement, a favorable mention. The excerpt from the students’ reading text below shows this infrequent meaning: […] But you can’t shake the feeling that it’s all just a big plug for Microsoft’s music store. […] The use of infrequent meanings of common words was key in this study. We intended to make the dictionary look-up process challenging for the participants. We wanted them to think that they knew or recognized a word, then discover upon reading a text that they might not know it. Using the dictionary appropriately would be of great importance if a student were to fully understand a given text. In general, while the participants were familiar with the most common meaning(s) of each word, they usually did not know or had only a vague comprehension of the infrequent meaning represented in the corresponding text. After the students read the text with the infrequent meaning, they were asked whether they knew the meaning of the word, and whether they could tell us the meaning. Then (regardless of their answers) they proceeded to look up the word. Students searched online for the meaning in the dictionary that resembled the meaning in the text. Next, the students were asked further questions, about whether their initial guess was correct, and how their initial guess compared to what they found in the dictionary (Topic 2: Look-up Ability of Participants). The next step was to ask the students questions related to the dictionary examples: about their usefulness, what the participant liked about them, and how they could be made better. One of these questions asked whether the example could be understood without information provided in square brackets “[ ]” (Topic 4: Perceptions of Usefulness and Quality of Dictionary Alenka Vrbinc, Donna M. Farina and Marjeta Vrbinc 94 Examples). Finally, participants were asked to explain what dictionary information was the most useful to them (Topic 3: Perceptions of Usefulness and Quality of Definitions). 5. Results Topic 1: Habits of dictionary use Our study found that most participants use a dictionary once per week or more; however, one participant uses dictionaries rarely and relies mostly on Google Translate. Those students who use dictionaries do not make discerning choices; they choose whichever online dictionary comes up first on a Google word search. None of the students had ever used the targeted MWLD, at least not that they were aware of. Eight of the nine participants were satisfied with the monolingual dictionaries that they use. The sole participant who expressed dissatisfaction was not discussing dictionaries but Google Translate (though this participant thought they were discussing dictionaries). All but one student stated that they find what they are looking for quickly and efficiently in dictionaries. Topic 2: Look-up ability of participants We mentioned earlier that while the students in this study were proficient in English, they were not English specialists (but rather specialists in some field of economics or business) and did not have the habit of attending to the finer points of language. Given this, it is not surprising that our participants often did not notice the part of speech of the target words as they read them in the short texts; they did not pay attention to whether a targeted word was a noun, verb, adjective, etc. So, when after reading they began looking up a word in the dictionary, the students continued not to notice part of speech. If we asked them, it was likely that the participants did know and could have told us the difference between, for example, a noun and a verb. However, during dictionary look-up, they were not alert to that quite significant difference and most of the time just ignored it. When the students read a word in a text or in the dictionary, they were interested solely in deciphering the meaning. Another problem related to the skill set of this group of students had to do with the form of the target word. It goes without saying that English words in real contexts, particularly verbs, do not always appear in the dictionary in their basic form. For example, a user will not find fixing (verb) at the head of a dictionary entry; instead, they will find fix and are expected to know the relationship between fixing and fix. Unfortunately, what lexicographers think users know is not always what they do know. This was often a factor leading to participants’ failure to find the correct meaning in Reflections on the Slovenian Dictionary User Study of 2018 95 the dictionary. All three of the verb forms in this study - taxed, fixing, and scoring - caused look-up problems for the users, and the form of the target verb was the main reason participants had difficulty locating the correct meaning. Seven of the nine participants did eventually find the correct meaning of the verbs used in their reading texts, but it was time-consuming and difficult for them. Topic 3: Perceptions of usefulness and quality of definitions One of our questions asked participants to indicate the dictionary information that was the most useful to them in understanding the meaning of a word. This question allowed us to glean information about students’ perceptions of definitions. For example, for the verb score, below is an excerpt from the reading text given to the students: Back when Patrick had a job at an auto-parts store and as a banquet server, his morning routine involved driving to Lawrence before work and scoring his daily fix. […] From the nine different meanings in the dictionary, eight of the nine students found the correct definition: slang: to buy or get (illegal drugs) Five of these eight mentioned the usefulness of the phrase illegal drugs in the dictionary definition. For the noun plug, as was noted above, students received the following reading text: […] But you can’t shake the feeling that it’s all just a big plug for Microsoft’s music store. […] One of the six different meanings given in the dictionary reads: something that is said on the radio, on television, etc., in order to create interest in something (such as a book, movie, or restaurant) - often + for Four of the eight students who found this correct meaning mentioned the grammatical information “often + for” as being very useful. It is probable that this information helped the students understand all three of the examples listed in the dictionary, all containing for: ‣ I heard a plug for that café on the radio. ‣ He gave a plug for [=talked about] his new film during the interview. ‣ She put in a plug for the band’s new album on her radio program. Alenka Vrbinc, Donna M. Farina and Marjeta Vrbinc 96 The perception of a participant that a certain dictionary definition was useful does not always mean that they selected the correct meaning. Sometimes a student who chose the wrong meaning still had a favorable view of the dictionary definition’s usefulness. Participants who picked an incorrect meaning provide us with indirect evidence that the definition might not have been so useful after all. Topic 4: Perceptions of usefulness and quality of dictionary examples We asked several different questions about the example sentences and phrases illustrating word use in the dictionary: what students liked about the examples, whether the examples conveyed the word’s meaning effectively, how the examples could be improved, whether material in square brackets “[ ]” within the examples was useful, and whether students would have understood an example without the material in square brackets. The students noticed and commented on the following aspects of dictionary examples: a) Length The participants stated most often that examples were too short, in particular when they were not in full sentences. In only a single instance during the entire study did a student note that an example was too long. For example, for the noun pitch, four of nine students said that the dictionary example an advertising pitch is too short. One student stated that they don’t like “short segments; ” another explained that the example “lacks context.” Another student considered the example too general: “You can advertise a lot of things; if you don’t know what pitch is, it’s not helpful.” b) Number of dictionary examples The students frequently expressed the desire for a greater number of examples. For example, as was mentioned above, the verb score had one meaning to buy or get (illegal drugs). This was illustrated by two example sentences: [+ object] ‣ He couldn’t score any drugs. [no object] ‣ Druggies come downtown looking to score. In all, for the nine different meanings of the verb score given in the dictionary, there are a total of 16 different examples. In general, the number of Reflections on the Slovenian Dictionary User Study of 2018 97 examples given throughout the online MWLD is relatively high. Nevertheless, the students in this study said that they wanted more. However, when they said they wanted a greater number of examples, it often meant that the examples which were provided did not cover their specific reading text. c) Similarity of wording in examples to wording in the reading texts The most difficult task for these participants was to connect the infrequent meaning of a word in their reading text to the correct related information in the dictionary. Students mentioned often the similarity or dissimilarity of their reading texts to the dictionary examples. In many cases where the researchers considered a dictionary example to be quite obviously similar to a reading text, the participants thought it was not similar. For example, the adjective sharp has one meaning stylish or fashionable in the MWLD. We provided the students with the following text related to this meaning: Her rock ’n’ roll friends might have expected a hip ’n’ cool outfit for her English country wedding. But it was her husband, Jamie Hince, the guitarist from The Kills, in his sharp blue Yves Saint Laurent suit, who brought a touch of musicworld fantasy. The MWLD provided the following three examples, two of which are full sentences and one a phrase: ‣ He’s a sharp dresser. ‣ a sharp outfit ‣ You’re looking very sharp today. While to the researchers these examples were analogous to the reading text that the students received, some of them saw things differently. Several students associated sharp with the color blue after reading the text, and even the above close dictionary examples did not help them grasp the correct meaning of sharp. d) Inclusion of verb forms in dictionary examples We have mentioned how some students struggled to look up verb forms such as fixing or taxed. Verb forms turned out to be relevant when students evaluated the dictionary examples. Participants firmly advocated for examples that would contain a variety of verb forms. And, they preferred to see the specific verb form that appeared in their reading text. For example, here is the reading text that the students received for fixing: Alenka Vrbinc, Donna M. Farina and Marjeta Vrbinc 98 ROVANIEMI, Finland—A man arrived at the police station here in 2011 with an unusual tip. He told the police that a Singaporean man was fixing matches with the local professional soccer team. The police were incredulous. Of the ten meanings of the verb fix in the MWLD, the correct one for the above text is: to control or affect (something, such as a game or election) in a dishonest way This definition was accompanied by the following examples: ‣ They were accused of fixing games in college. ‣ The election was fixed. ‣ fix a parking ticket [=arrange for someone to not have to pay a fine for parking illegally] One of the examples does have the same form, fixing, that is found in the students’ text. The examples also cover two other forms, fixed and fix. While these are indeed the types of examples students said they liked, if the form of the verb was so misleading to begin with that students could not find the correct meaning, then a good example (from their perspective, such as the one with fixing games above) would still not help them. e) Information in square brackets within examples The majority view among the students was that they preferred having additional information within the dictionary examples, of the type that is usually given in square brackets “[ ]” in the MWLD. For the verb tax, the dictionary listed the following examples, with supplementary phrases in square brackets: ‣ That job really taxed our strength. [=required us to use a lot of physical effort] ‣ All this waiting is taxing my patience. [=is making me lose my patience] ‣ puzzles that tax your brain ‣ You can have an enjoyable vacation without taxing your budget. [=without having to spend a lot of money] One student said: “Additional explanations are helpful because the words are explained once again in a simple way.” For the fourth example above, this same student said that the material in brackets was essential for their understanding. Reflections on the Slovenian Dictionary User Study of 2018 99 f) Use of boldface and italics in dictionary examples Our students were quite sensitive to the use of boldface and italics; their level of awareness was a surprise in the study. Some students said that they did not understand the difference between the use of bold and italics; one student made several guesses, all incorrect, as to what the difference was. Another student said that way the dictionary uses both italics and boldface is “inconsistent.” At times, students expressed appreciation for boldface, because for them it made some information more prominent (cf. Herbst 1996 and Dziemianko 2014 on highlighting word combinations in boldface). For the dictionary entry of the noun plug, one student said (as a compliment) that the boldfaced text “sticks out.” On the other hand, all the students who chose to comment on the issue conveyed their dislike of italics. One student called italics “clutter.” 6. Discussion The answers to questions related to habits indicated that the student-participants use dictionaries, although not as often as lexicographers might like. Participants’ lack of extensive dictionary experience meant that their look-up skills were often not strong enough for them to benefit fully from their interaction with the dictionary. While their look-up problems were sometimes interrelated and (for example) two problems together could contribute to a prolonged or unsuccessful look-up, even one problem by itself could cause a user to make mistakes that cost them time and/ or led to incorrect identification of the meaning. Our participants pointed out that, while dictionaries are created by experts, they can be improved if dictionary makers “can learn something about average users; ” if linguists can “think like the average person.” What follows are recommendations for those involved in learner lexicography. Improving part-of-speech information to make look-up easier During the look-up activities, some participants noticed (and used) the MWLD’s small drop-down menu with parts of speech labeled - and others never did. Those who did use this menu said that it made look-up easier. The drop-down menu as it is currently presented in the online MWLD is not as salient, and thus not as useful as it could be in guiding users to the right part of speech. Among the target words of this study, the most blatant case of the menu being unhelpful is the MWLD’s treatment of mean (see Figure 1), a word than can be used as a noun, verb, or adjective. The first three choices in the seventeen-item drop-down menu are: mean (verb), mean (adjective), and mean (adjective). A user has no way of understanding which of the two visible adjective listings should be selected. Moreover, Alenka Vrbinc, Donna M. Farina and Marjeta Vrbinc 100 since only three items are visible at a time in this menu, one must use the scrolling arrow on the right to view the additional fourteen choices. Figure 1. Drop-down menu for mean. Improving dictionary examples Our participants wanted full-sentence examples even in cases where most lexicographers would regard a brief phrase as enough. The participants appeared to be saying that it would be better to err on the side of longer rather than shorter examples. The participants’ desire to have the example most similar to their reading text is related to their interest in a larger quantity of examples. Learners have more of a possibility to find the example that works for them if there are more examples. If a dictionary were to provide more examples, then more verb forms such as fixing and taxed could figure within the full series of examples provided; this was another clear desire of the students studied. While the study participants said that they wanted more examples, they also said they do not want too many; this is not as contradictory as it seems. One student suggested the possibility of clicking to receive more examples when desired, or the ability to click on a short example to access a sentence or paragraph. This participant believed that dictionary users should be able “to go deeper” into the information, but that those who do not wish to do so should not be “force[d] … to read everything” on the web page. Reflections on the Slovenian Dictionary User Study of 2018 101 Improving the way dictionary information is presented One participant suggested the use of color to clarify the presentation of information: Different types of information could be in different colors. This idea was actually used in the 2008 print edition of the MWLD, where examples appeared in blue font and definitions in black. For unknown reasons, the use of color did not transfer over to the MWLD online, where the font for examples is now black. Bringing blue back into this dictionary might help learners distinguish between the definition and the examples; adding other colors might further improve the look-up process. One unequivocal message from the students was their aversion to italics. They did not understand why italics were used in the dictionary or how their meaning differs from that of boldface. The participants thought italics caused text to be “unnoticeable; ” they were “old fashioned.” As non-native speakers albeit advanced learners of English, it is possible that italicized text presented comprehension problems for the students in this study. Many sources advise against using italics in readings meant for dyslexic people. 5 The researchers speculate that, in the online environment in which the study participants do most of their reading, italicized text is indeed less visible and prominent. The online environment has multiple means to render text salient, so italics do not appear to be a good choice there, given how poorly they render on a screen. The students said they liked boldface, although their remarks indicated an incomplete or nonexistent understanding of how the dictionary used it. One student (incorrectly) believed that boldface was how the dictionary shows “the reader which example is more appropriate.” 7. What are the takeaways for lexicographers? The first takeaway from this study is that while education in dictionary use is important, it alone will not suffice to increase dictionary use or the understanding of dictionaries among the younger generations. While dictionary instruction is carefully embedded in the formal curriculum in Slovenia, in practice numerous students report that no instruction or very little instruction in dictionaries actually takes place. This might be the reason that one of our participants thought they were using a dictionary (a bad and dissatisfying one) when they were really accessing Google Translate. As dictionaries have moved online, they have lost their distinctness from other available internet tools. And, that distinctness has turned out to be important: In the print era, people had an emotional attachment to their specific print dictionary and were concerned about the correct definitions of words. Today, an online dictionary is just another online resource; it is 5 For example, the British Dyslexia Association (n.d.) advises: “Avoid […] italics as this can make the text appear to run together […] Use bold for emphasis”. Alenka Vrbinc, Donna M. Farina and Marjeta Vrbinc 102 there to take or leave. It is certainly the case that current students are completely unaware of the correct way to use learner’s dictionaries - and prefer using mobile phone dictionary apps. Even if further education were provided, it is doubtful that most students would go out of their way to use one online dictionary over another, to compare online dictionaries and decide which they prefer. People have become used to the nondescriptness of the online environment: You type something and go where you are taken by search engines. Using such a powerful and sophisticated tool - as an online learner’s dictionary is - requires habit formation as well as training. A reliable dictionary is better and more authoritative than any random app and using such a dictionary to its full potential would allow users to avoid frustration and obtain more information. This constitutes not only a takehome message for the lexicographers about what users need and want, but it is also a plea for language instructors to do more than pay lip service to teaching students how to use legitimate lexicographic tools properly, for best results. The second takeaway from the Slovenian study is related to the first. While online dictionaries may be perceived as interchangeable by the user, they could - and should - become less nondescript than they are. The online Merriam-Webster Learner’s Dictionary, as a continuation of the 2008 print Merriam-Webster’s Advanced Learner’s English Dictionary, was not wellserved by its crossover to the online environment. The inadequate dropdown menu and the lack of color to distinguish some parts of the entry (something that exists in the printed book) are only two of several areas in need of improvement. As an online tool, the MWLD is less useful than it could be and does not fulfill the promise of the online environment. In other spheres of its endeavors, the Merriam-Webster company deserves praise for rendering its dictionaries more distinct and more significant to the public. The company and its lexicographers are often in the news in the U.S. because of their quality reporting on words that are trending in online dictionary look-ups. For example, on September 10, 2021, look-ups of the word mandate spiked 500%, as Merriam-Webster maintains, “following President Biden’s speech on new efforts to increase rates of vaccination.” 6 Merriam’s postings on trending words are in sync with high-interest social and political events and demonstrate unequivocally that words - and dictionaries - matter. On the other hand, the potential of the Merriam learner’s dictionary is not being realized. The participants in our study had high expectations of how the online dictionary environment should look, what types of information should be in it, and how they wanted that information delivered to them. They reported to us that, prior to taking part in the Slovenian study, their own look-up efforts sometimes ended in dissatisfaction. Certainly, during the 6 https: / / www.merriam-webster.com/ news-trend-watch/ biden-vaccine-mandates- 20210910. Reflections on the Slovenian Dictionary User Study of 2018 103 study they demonstrated a great sensitivity to a variety of dictionary elements and sometimes expressed discouragement with what they found. Nevertheless, the students said that they enjoyed using the online Merriam- Webster Learner’s Dictionary overall; many of them planned to continue using it. Their perceptions are valuable to lexicographers who aim to improve the content, format, and presentation of information in learner’s dictionaries. The Slovenian study is a reminder that the stakes are high; the dictionary needs to stand out amid the pervasive noise that the online environment represents. Acknowledgements An earlier version of this paper, “Reflections on the Slovenian User Study: What Does It Tell Us? ” was read at the October 2019 EMLex Autumn Meeting and Colloquium, held by the Lexicographic Centre, Ivane Javakhishvili Tbilisi State University, Tbilisi, Georgia. The authors acknowledge the grant project, Dictionary User Groups: What They Can Teach Lexicographers (BI-US/ 17-18-033), which was financially supported by the Slovenian Research Agency. They also acknowledge the approval (2 March 2018) of the New Jersey City University (NJCU) Institutional Review Board for the Protection of Human Participants in Research, and the approval (5 March 2018) of the Ethics Committee of the Faculty of Arts, University of Ljubljana. Donna Farina thanks NJCU for travel support to Ljubljana, Slovenia. Last and most importantly, the authors are very grateful to the study participants from undergraduate programs in the School of Economics and Business at the University of Ljubljana. References A. Dictionaries Merriam-Webster Learner’s Dictionary. http: / / learnersdictionary.com/ . (MWLD) B. Other literature Atkins, Beryl T. Sue (Ed.) (1998). Using Dictionaries: Studies of Dictionary Use by Language Learners and Translators (Lexicographica, Series Maior 88). Tübingen: Max Niemeyer Verlag. Atkins, Beryl T. Sue & Michael Rundell (2008). The Oxford Guide to Practical Lexicography. Oxford: Oxford University Press. Barnhart, Clarence L. (1962). Problems in editing commercial monolingual dictionaries. In: Fred W. Householder & Sol Saporta (Eds.). Problems in Lexicography. Bloomington: Indiana University. 161-181. Alenka Vrbinc, Donna M. Farina and Marjeta Vrbinc 104 Battenburg, John D. (1994). Pioneer in English lexicography for language learners: Michael Philip West. Dictionaries: Journal of the Dictionary Society of North America 15: 132-148. https: / / doi.org/ 10.1353/ dic.1994.0017 Béjoint, Henri (1994). Tradition and Innovation in Modern English Dictionaries. Oxford: Clarendon Press. Béjoint, Henri (2010). The Lexicography of English. Oxford: Oxford University Press. British Dyslexia Association (n.d.). Creating a dyslexia friendly workplace. [online] https: / / www.bdadyslexia.org.uk/ advice/ employers/ creating-a-dyslexia-friend ly-workplace/ dyslexia-friendly-style-guide [December 2021]. Collins English Language Teaching Blog (n.d.). [online] news.collinselt.com/ cobuilddictionary-online/ [January 2022]. Cowie, Anthony P. (Ed.) (1987). The Dictionary and the Language Learner: Papers from the EURALEX Seminar at the University of Leeds, 1-3 April 1985. Tübingen: Max Niemeyer. https: / / doi.org/ 10.1515/ 9783111340500 Cowie, Anthony P. (1998). A.S. Hornby, 1898-1998. A centenary tribute. International Journal of Lexicography 11 (4): 251-268. Cowie, Anthony P. (1999). English Dictionaries for Foreign Learners: A History. Oxford: Oxford University Press. De Schryver, Gilles-Maurice & Danie J. Prinsloo (2011). Do dictionaries define on the level of their target users? A case study for three Dutch dictionaries. International Journal of Lexicography 24 (1): 5-28. https: / / doi: 10.1093/ ijl/ ecq045 Dolezal, Frederic T. & Don R. McCreary (1999). Pedagogical Lexicography Today: A Critical Bibliography on Learners’ Dictionaries with Special Emphasis on Language Learners and Dictionary Users. (Lexicographica, Series Maior 96). Tübingen: Max Niemeyer. https: / / doi.org/ 10.1515/ 9783110947014 Dziemianko, Anna (2014). On the presentation and placement of collocations in monolingual English learners’ dictionaries: Insights into encoding and retention. International Journal of Lexicography 27 (3): 259-279. https: / / doi.org/ 10. 1093/ ijl/ ecu012 Farina, Donna M. T. Cr., Marjeta Vrbinc & Alenka Vrbinc (2019). Problems in online dictionary use for advanced Slovenian learners of English. International Journal of Lexicography 32 (4): 458-479. https: / / doi.org/ 10.1093/ ijl/ ecz017. Gouws, Rufus H. (2014). Article structures: Moving from printed to e-dictionaries. Lexikos 24: 155-177. https: / / doi.org/ 10.5788/ 24-1-1256 Hannabuss, Stuart (1996). Research interviews. New Library World 97 (5): 22-30. https: / / doi.org/ 10.1108/ 03074809610122881 Hatherall, Glyn (1984). Studying dictionary use: Some findings and proposals. In: Reinhard Rudolf K. Hartmann (Ed.). LEXeter ‘83 Proceedings. Tübingen: Max Niemeyer. 183-189. Herbst, Thomas (1996). On the way to the perfect learners’ dictionary: A first comparison of OALD5, LDOCE3, COBUILD2 and CIDE. International Journal of Lexicography 9 (4): 321-357. https: / / doi.org/ 10.1093/ ijl/ 9.4.321 Kosem, Iztok et al. (2019). The image of the monolingual dictionary across Europe. Results of the European survey of dictionary use and culture. International Journal of Lexicography 32 (1): 92-114. https: / / doi.org/ 10.1093/ ijl/ ecy022. Landau, Sidney ([1984] 2001). Dictionaries: The Art and Craft of Lexicography. Cambridge: Cambridge University Press. Lew, Robert (2011). Space restrictions in paper and electronic dictionaries and their implications for the design of production dictionaries. [online] https: / / repozytorium.amu.edu.pl/ bitstream/ 10593/ 799/ 1/ Lew_space_restrictions_in_paper_ and_electronic_dictionaries.pdf [Feb. 2022]. Reflections on the Slovenian Dictionary User Study of 2018 105 Lew, Robert & Gilles-Maurice de Schryver (2014). Dictionary users in the digital revolution. International Journal of Lexicography 27 (4): 341-359. https: / / doi. org/ 10.1093/ ijl/ ecu011 Liu, Xiqin, Dongping Zheng & Yushuai Chen (2019). Latent classes of smartphone dictionary users among Chinese EFL learners: A mixed-method inquiry into motivation for mobile assisted language learning. International Journal of Lexicography 32 (1): 68-91. https: / / doi.org/ 10.1093/ ijl/ ecy019 Lorentzen, Henrik & Liisa Theilgaard (2012). Online dictionaries - how do users find them and what do they do once they have? In: Ruth Valtvedt Fjeld & Julie Matilde Torjusen (Eds.). Proceedings of the 15 th EURALEX International Congress. Oslo: Department of Linguistics and Scandinavian Studies, University of Oslo. 654-660. Müller-Spitzer, Carolin, Alexander Koplenig & Sascha Wolfer (2018). Dictionary usage research in the Internet era. In: Pedro A. Fuertes-Olivera (Ed.). The Routledge Handbook of Lexicography. Milton Park, New York: Routledge: 715- 734. Qu, Sandy Q. & John Dumay (2011). The qualitative research interview. Qualitative Research in Accounting & Management 8 (3): 238-264. https: / / doi.org/ 10.1108/ 11766091111162070 Rundell, Michael (1998). Recent trends in English pedagogical lexicography. International Journal of Lexicography 11 (4): 315-342. Rundell, Michael (2012). Stop the presses: The end of the printed dictionary. Macmillan Dictionary Blog. [online] http: / / www.macmillandictionaryblog.com/ bye-print-dictionary [December 2021]. Svensén, Bo (2009). A Handbook of Lexicography: The Theory and Practice of Dictionary-making. Cambridge: Cambridge University Press. Tomaszczyk, Jerzy (1979). Dictionaries: Users and uses. Glottodidactica 12: 103- 19. Vrbinc, Marjeta, Donna M. T. Cr. Farina & Alenka Vrbinc (2022). Who is sharpest at looking up sharp? Comparing two parallel groups of dictionary users. Lexikos 32: 368-391. https: / / doi.org/ 10.5788/ 32-1-1750. Vrbinc, Marjeta, Alenka Vrbinc & Donna M. T. Cr. Farina (2023). The lexicographer’s dream audience: Dictionary use among English majors at a Slovenian university. International Journal of Lexicography 36 (2): 149-172. https: / / doi. org/ 10.1093/ ijl/ ecac019 Wingate, Ursula (2002). The Effectiveness of Different Learner Dictionaries. Tübingen: Max Niemeyer. Alenka Vrbinc University of Ljubljana Donna M. Farina New Jersey City University Marjeta Vrbinc University of Ljubljana