AutoCWW Tutorial
Transcription
AutoCWW Tutorial
Steps of the Cognitive Walkthrough for the Web (CWW) Navigation System Analysis Table of Contents Part 1. Web-based interface . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Part 1. Readability Evaluation Tool . . . . . . . . . . . . . . . . . . . . 3 Part 3. LSA measures and state-of-the-art parameters . . . . 3 Part 4. Formulate set of user goals . . . . . . . . . . . . . . . . . . . . 4 Part 5. Example of web-based CWW for real website. . . . . . 5 Part 6. Compute predicted mean total clicks for each user goal, and test predictions in the lab . . . . . . . . . . . . . . . . . . .14 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 Appendix A: Elaborated Encarta headings and links . . . . .18 Appendix B: Sample user goals for Online Encyclopedia .29 2 Steps of the Cognitive Walkthrough for the Web (CWW) Navigation System Analysis 1 Web-based interface This tutorial assumes that the reader is familiar with Latent Semantic Analysis and with the Cognitive Walkthrough for the Web (CWW). If not, the reader can read the papers listed in the References section, track and download LSA papers at http://lsa.colorado.edu, and track and download papers on CWW and ten experiments to date validating CWW via http://AutoCWW.colorado.edu/~blackmon/. The server name, AutoCWW (see Figure 1) indicates the long-range goal of the CWW research group to incrementally automate the CWW procedures to make them more efficient for researchers, HCI students – and, ultimately, collaborating web designers/developers, particularly those engaged in building informational websites that offer medical/health and scientific information to the general public. Figure 1 shows the tools that are available on the web-based interface as of July 2004. In the Figure 1. Home page of http://AutoCWW.colorado.edu, showing tools available in July, 2004. This home page appears immediately after login at http://AutoCWW.colorado.edu. Created July, 2004 3 sections that follow, we will identify available tools by the name shown in Figure 1, but – a word of caution – the web-based interface at http://AutoCWW.colorado.edu may be updated with additional tools and help information before the next update of this tutorial. 2 Readability Evaluation Tool The Readability Evaluation Tool is designed as a plug-in to Microsoft Word for Windows, appearing under “Tools” on the Word menu. Both the Readability Evaluation Tool and the accompanying manual for installing and using the tool can be downloaded at http://AutoCWW.colorado.edu/Readability/ReadabilityTool.html. 3 LSA measures and state-of-the-art parameters 3.1 Similarity The Latent Semantic Analysis (LSA) measure of similarity – the cosine of the angle between the vectors of two texts – is the LSA measure that is best known and most widely used. CWW employs the LSA cosine as a measure of the information scent between a given user goal text and each heading text and link text on a particular webpage. • Weak scent is a goal-link cosine for the correct link with a cosine <0.10 • Confusable heading/link texts are defined as a pair of texts with a cosine !0.60 The LSA cosine is also used by the Readability Evaluation Tool to assess appropriateness of the content webpages for a particular target audience of the website, setting the following parameters for the cosine values: • Sentence-to-sentence coherence: minimum mean cosine of 0.35 • Key concepts in comparison with relevant text delineating the key concepts: !0.60 3.2 Familiarity LSA offers two measures of familiarity: (1) frequency of a particular word within a semantic space designed by scientifically sampling documents that have typically been read by a particular user group, and (2) term vector length of a word or phrase within such a semantic space. Term vector length measures background knowledge related to a topic (e.g., archaeology) for the group of users represented by the semantic space. A user must know the meanings of the words in the heading and link texts in order to perceive information scent when the headings/link texts are highly similar to the user’s goal. Therefore, information scent is diminished if a heading/link text consists exclusively, or even partially of low frequency words. Information scent is also reduced if the percentage of low frequency words is too high in a particular subregion of the webpage, making it impossible to infer the probable meaning of the low frequency words from the surrounding context of familiar words. Here are the currently values of the relevant parameters: • Word frequency !15 in the selected semantic space is required for the word to be considered sufficiently familiar, else the word is low frequency and unfamiliar • Percentage of low frequency words damages comprehension of a content article if above 8% low frequency words – measured by dividing the number of unique low frequency Created July, 2004 4 words (words with a frequency of <15 in the semantic space) by the total number of unique words in the text. • Term vector length >0.55 for 1-word link/heading, else !.80. Term vectors below these one-word and multiple-word thresholds are considered to imply insufficient background knowledge of the topic. 3.3 Similarity and familiarity simultaneously measured To simulate text elaboration during comprehension we assume that the words that are activated in the user’s memory to elaborate the printed text are words that are both highly familiar and highly similar to the printed text being read by the user. • “Highly familiar” is measured using a minimum word frequency of 50 in the semantic space • “Highly similar” to the text being read is measured by using a minimum cosine of 0.50. 4 Formulate set of user goals To design or re-design an actual website, the ideal is to interview prospective users of the website to discover their goals for using the website. For example, the Osteosarcoma Online website uses interviews of, and feedback from, osteosarcoma patients, members of patients’ families, and friends of osteosarcoma patients. The goal is to discover the information that these users want to find on the website and to make sure the information is provided at the right level of detail and level of general reading knowledge and background knowledge of medicine. 4.1 List user groups and semantic space that most validly represents each user group If experiments are to be run using American undergraduate students as experimental participants, then the most appropriate semantic space is general reading knowledge of first-year college (tasaALL). Other general reading knowledge semantic spaces are available for English-speaking Americans at 3rd, 6th-, 9th-, and 12th-grade reading level. There is also a semantic space for college-educated French language speakers, based on a corpus of a full year of Le Monde newspaper. Additional semantic spaces are in preparation but not yet available. 4.2 Representative set of user tasks for each user group So far, CWW has been applied to tasks where the user is attempting to find information on a website, such as an online encyclopedia, a medical/health website for patients and other lay users, or a website that provides scientific information. 4.3 Discount list of user tasks: full text or summaries of all content webpages The user goals that have been used in experiments by members of the CWW research team are generally summaries of content articles available on the existing website being evaluated, using these criteria: • Each summary presented to experimental participants is typically 100-200 words. Created July, 2004 5 • In order to ensure that the summary accurately represents the full article, there must be a cosine of 0.80 or higher between the summary and the full article text. Microsoft Word contains a summarization tool that produces a first draft of the summary in complete sentences. The draft sometimes needs to be hand edited for coherence or to increase the cosine between the summary and full text of the article to the threshold value of 0.80 or higher. 5 Example of web-based CWW for real website This example is designed to make predictions about user performance on one webpage, distinguishing which tasks will be easy to do and which tasks will encounter significant usability problems. The example webpage, shown in Figure 2, is the home page of Encarta Online Encyclopedia (http://encarta.msn.com). These predictions have evolved from laboratory experiments that use simulations of this webpage with college students. Extrapolating from accumulated data for simulations of this webpage, we can predict the approximate mean number of clicks (to an estimated accuracy of ± 2 clicks) that college students require to complete the task. The measure “Mean Total Clicks” is a measure of problem severity that turns out to be closely correlated with another measure of problem severity, namely the approximate rate of task failure, i.e., percentage of users who will not be able to complete the task in 130 seconds or less. To keep this example as simple as possible we will also abide be the following constraints: Figure 2. Encarta online encyclopedia showing nine categories that house a total of 93 links Created July, 2004 6 • Restrict attention to links and headings in the content area of the webpage • Group in a simple matrix, ignoring effects of graphic design elements on user behavior • Postpone detailed descriptions of CWW repair methods and their success rates. 5.1 Identify visuospatial-semantic groupings The simplest approach applies CWW just to the headings and links in the content area of a single webpage, allowing us to make and test predictions for tasks that can be accomplished with a minimum single click on one webpage. For this example we will use CWW to predict human performance finding encyclopedia articles on specific topics on the Encarta Online Encyclopedia, a webpage that offers 93 links nested under nine different categories/headings. Figure 2 shows the webpage as it first appears to users. Clicking on any of the nine categories reveals the links nested under that one category heading, and white space separates groupings. Figure 3 shows the webpage used in laboratory simulations of this experiment, displaying all 93 links in a simple three-by-three matrix with nine cells, one for each heading. Comparing Figures 1 and 2, we see that the actual webpage displays a few links that do not appear in the simulated version. The top navigation bar in Figure 2 presents links to Home (home of the Microsoft reference center), Encyclopedia (present location, the main page of a major subsite of the reference center), Atlas, Dictionary, Multimedia, Magazines, Homework, Math Help, and Search Encarta. None of these additional links, however, are relevant to the task of browsing the online Figure 3. The task Find encyclopedia article about Hmong in the simulation of the Encarta online encyclopedia with 93 total links nested within a matrix of nine headings Created July, 2004 7 encyclopedia to find an article on a particular subject. (In order to focus exclusively on finding articles by browsing we exclude the site search engine.) Therefore, the simplified matrix presented in Figure 3 captures the essence of search tasks on the encyclopedia. College students who participated in a CWW experiment did a series of pages that all looked like Figure 3 except for variations in the orange box telling participants what to search for. The Figure 3 task is “Find an encyclopedia article about Hmong.” Other tasks searched for such articles as Music Therapy, Cybernetics, and Ferreting. The links nested under each of the nine categories are listed in exactly the same order that they appear on the webpage. The visuospatial grouping in Figure 3 is a matrix design with borders outlining each matrix cell to clearly distinguish each cell from other cells. Headings stand out in contrasting color font, larger size font, and boldface for emphasis. 5.2 Identify insufficient familiarity problems To accurately predict human performance on webpages one necessary step is to identify heading and link texts that will probably be unfamiliar for a target group of website users. For experiments, the goal texts should also be analyzed to ensure that they do not contain a lot of Figure 4. Encarta heading and link texts ready to submit to Low Frequency Words Analysis in the semantic space for third-grade general reading knowledge Created July, 2004 8 unfamiliar words and concepts and, thus, are not difficult for experimental participants to comprehend. There are two LSA-based measures of familiarity. The first is identification of all the low frequency words and computation of the percentage of low frequency words. The second is finding link texts with low term vector length. The Readability Evaluation Tool (plug-in to Microsoft Word for Windows) provides the most efficient way to identify low frequency words, compute the percentage of low frequency words, and edit to reduce the percentage. Alternatively, use the web-based tool by clicking the link “Low Frequency Words Analysis” on the home page shown in Figure 1. Figure 4 gives a snapshot of the text submitted to Low Frequency Words Analysis as well. To find exact frequencies of words, submit the words to the “Frequency Analysis” tool. Because the Frequency Analysis tool reports the frequency of each unique word only once, it is possible, but timeconsuming and tedious, to use the “Frequency Analysis” tool to compute word frequency for each and every word in a text, sort the list by word frequency, and then compute the percentage of low frequency words by dividing the unique number of low frequency words by the total number of unique words in the passage. By whatever method we choose to do the computation, we can discover that there are 154 unique words in the nine headings and 93 links shown in Figure 3. Of these 154 words, seven (5%) are low frequency words even in the first-year college semantic space (e.g., Australasia, paleontology, archaeology, monerans, and occult). Low frequency words cause avoidable Figure 5. How third graders see the task “Find encyclopedia article about Hmong,” showing low frequency words in red font and low term vector heading/link texts in bold italics Created July, 2004 9 usability problems even at the 5% level, but users with third-grade reading level face 51% low frequency words (79/154) on the same page, an overwhelming source of usability problems. For users with sixth-grade reading level, 18% of the words are low frequency (27/154). On webpages, users tend to ignore links that are unfamiliar, because the correct link has little or no information scent for a user who has rarely or ever encountered the words in the link. Figure 5 shows all the usability problems that would face a third grader trying to find an article in the Encarta encyclopedia. Red font indicates words that have low frequency in the third-grade semantic space, and bold italic font indicates headings and links that have low term vector length. Four out of nine headings (44%) have low term vector length, and 51 out of 93 links (55%) have low term vector length. Third-graders could probably find the familiar links “Fish” or “Birds” or “Plants” under the familiar heading “Life Science,” but third-graders might never find the familiar link “Music” under the unfamiliar heading “Performing Arts” or click any link under the unfamiliar heading “Religion & Philosophy.” A high percentage of low frequency words makes it difficult to infer or guess the meaning of an unfamiliar word from the surrounding context, because the surrounding context is also full of unfamiliar words. Even when a link text does not use any low frequency words, however, the user may still not have sufficient background knowledge of the topic. The term vector length measures the degree to which the semantic space contains information closely related to a heading or link text. In the 10 experiments run to date by CWW researchers, there has been a consistent result that identifies low background knowledge – as measured by term vector length – as a serious impediment for users searching for information that requires clicking an unfamiliar link in order to find it. Table 1 displays the output from the AutoCWW Unfamiliar headings/links analysis for the 93 links on the Encarta.msn.com website. This analysis identifies as unfamiliar any link that is Table 1. Unfamiliar links in college-level semantic space for Encarta Online Encyclopedia (low frequency words in red font, word frequencies and one-word term vector lengths in parentheses) One-word links Term vector length !0.55 Links with two or more words Term vector length <0.80 Paleontology (4) 0.06 Legends (96) & Folklore (21) 0.39 (0.32, 0.11) The Occult (11) 0.08 Anatomy (76) & Physiology (48) 0.51 (0.30, 0.24) Scripture (16) 0.10 Theology (31) & Practices (456) 0.76 (0.20, 0.67) Archaeology (11) 0.10 Calendar (143), Holidays (118), & Festivals (43) 0.77 (0.49, 0.28, 0.23) Mythology (38) 0.17 Anthropology (65) 0.50 Pets (199) 0.51 Communications (213) 0.55 <0.80, but a better parameter for one-word links appears to be "0.55. Term vector lengths for two or more words are always longer than the term vector length for single words, so it is more Created July, 2004 10 reasonable to have separate parameters for one-word heading/link texts and for heading/link texts of two or more words. There is no quick fix for lack of background knowledge. Repairing unfamiliar links can be done by altering the heading/link text, e.g., changing the link “Paleontology” to “Paleontology and Fossils,” or changing the link “Anthropology” to “Anthropology and Cultures.” Such changes can help when the link text uses low frequency words, but they cannot genuinely compensate for insufficient background knowledge. Another repair provides an alternative path to the information by clicking a familiar link that is highly similar link to the user's goal. Figure 6. Set of raw link texts ready to submit for elaboration with a minimum frequency of 50 and minimum cosine of 0.50. Created July, 2004 11 Table 2. Selected raw link texts and their elaboration by AutoCWW “Elaborate Links” 5.3 Elaborate link and heading texts to simulate reading comprehension The reading comprehension process elaborates the text that appears on the page by activating words that are semantically similar to the text being read (Kintsch, 1998). Activation of semantically similar words also accounts for the rapidity with which people learn new vocabulary, because learning or improving learning of one new word simultaneously adds to the knowledge of semantically related words (Landauer & Dumais, 1997). Selecting the link “Elaborate Links” on the AutoCWW home page (Figure 1) makes it possible to submit a whole set of raw link/heading texts for LSA Near Neighbors analysis and to set the parameters. Figure 6 shows a set of raw text links ready to be submitted. Since LSA Near Neighbor analyses are time-consuming, it helps to not submit more than 50 raw texts at one time (to avoid browser timeouts). The semantically related words most likely to be activated are words that are both highly familiar and semantically close, so we recommend setting the minimum word frequency at 50 and the minimum cosine at 0.50, as illustrated in Figure 6. Well-designed raw link texts use a minimum number of words and occupy a minimum amount of screen space but use words that are rich in meaning, activating many other semantically related, familiar words. Table 2 displays a sample of raw text links and the elaborated texts output from the “Elaborate Links” tool. As Table 2 demonstrates, even a single word can activate a host of related words in the mind of the user reading a one- or two-word link. To capture the complex meaning of a heading and the cluster of links nested under the heading, our current practice is to elaborate each heading text and add to it the elaborated texts of all the subordinate links, creating a large cluster of words. For example, the heading “Performing Arts” and the elaborated versions of its six subordinate links – “Music,” “Musicians & Composers,” “Musical Instruments,” “Dance,” “Theater,” and “Cinema, Television, and Broadcasting” – becomes the lengthy elaborated heading text below: Performing Arts arts art artistic artists painters sculpture sculptures architecture Theater theater actors actor playwright playwrights drama plays theatre comedy shakespeare costumes performers broadway audiences scenery macbeth tragedy scenes theaters performances stage staged elizabethan script theatres scene audience costume performer performed movie chorus Musicians & Composers composers musicians jazz music singers beethoven musician orchestra musical melody piano tunes symphony concert songs blues opera chord dance song rhythmic sing rhythm concerts dancing singing tune singer violin dancer guitar rhythms performances danced instruments sung dancers ballet sang tones instrument band drums chorus dances chant folk choir instrumental strings Cinema, Television, & Broadcasting television viewers broadcast broadcasting tv commercials radio stations entertainment media movies newspapers network news magazines advertisers networks radios communications studio appearances channels Music music jazz singers musicians songs composers tunes musical melody piano orchestra beethoven musician concert song blues symphony sing opera rhythm singing tune dance rhythmic singer concerts guitar sang dancing rhythms drums sung danced chord violin choir band chant dancer performances dancers ballet instruments dances folk chorus flute tones strings instrument harp lessons sings Dance dance dancing ballet dancer dancers dances music danced jazz musicians piano composers concert orchestra singers musician opera tunes concerts performances beethoven songs melody blues musical performers sing symphony costume Musical Instruments instruments musical instrument music jazz melody piano musicians musician guitar strings violin orchestra songs concert rhythmic tune rhythm blues Raw Encarta link texts Elaborated text, minimum frequency 50, minimum cosine 0.50 Created July, 2004 12 Artists artists art painters artist paintings painting painter artistic sculpture portraits sculptures painted paint portrait architecture museum museums realism renaissance decorative arts sculptor patrons paints gallery statues florence claude geometric style Theater theater actors actor playwright playwrights drama plays theatre comedy shakespeare costumes performers broadway audiences scenery macbeth tragedy scenes theaters performances stage staged elizabethan script theatres scene audience costume performer performed movie chorus Music music jazz singers musicians songs composers tunes musical melody piano orchestra beethoven musician concert song blues symphony sing opera rhythm singing tune dance rhythmic singer concerts guitar sang dancing rhythms drums sung danced chord violin choir band chant dancer performances dancers ballet instruments dances folk chorus flute tones strings instrument harp lessons sings Language language spoken dialect languages dialects speak grammatical vocabulary grammar expressive speaking slang english Birds birds bird feathers beaks beak wings eagle nest nests loon fly nesting hummingbirds wing robins geese hawk flew pigeons feather gull crows eagles cranes ostrich swan hummingbird pigeon owls fluttered flying Medicine medicine medicines doctor doctors prescription sick medical clinic Physics physics physicists science sciences physicist biology geology chemistry einstein scientific mathematics Anthropology anthropology anthropologist sociology anthropologists disciplines humankind sociologists sciences mead societies human beings culture scientifically psychology cultures Scripture sin, theology Countries countries underdeveloped industrialized nations 5.4 Identify goal-specific competing headings and links The full set of elaborated heading and link texts for the Encarta encyclopedia is shown in Appendix A, ready to copy and paste into AutoCWW “One-to-Many Analysis” for comparison with each specific user goal. Sample user goals for Encarta Online Encyclopedia are shown in Appendix B. Created July, 2004 13 Figure 7 shows the One-to-Many webpage of AutoCWW with one goal – “Find an encyclopedia article about Hmong” – in the upper box and a large set of nine elaborated heading texts and 93 elaborated link texts ready to submit for comparison with the goal. As specified on the website, an empty line separates each elaborated text from the adjacent texts above and below it. One-toMany returns a two-column table of results with rows for each heading and link text. Column one lists each one of the texts, and column two displays the cosine between each text and the goal statement. The next step is to copy that table of results and paste it into an Excel spreadsheet. Our practice is to copy the results into the A and B column of an Excel sheet, specify whether the text is a Figure 7. AutoCWW One-to-Many Comparison webpage forms ready to submit for comparison of the Find Hmong goal with each Encarta heading and link text, using the college-level space Created July, 2004 14 heading or a link text in column C, and then indicate the specific parent heading for each link in column D (e.g., the parent heading “Performing Arts” for the link “Music”). That four-column table is then copied and pasted in columns F-G-H-I, and then sorted first by heading or link and then by descending cosine value. The “correct” heading(s) and link(s) are highlighted in green. Goal-specific competing headings are headings with cosine values equal to or greater than the cosine of the “correct” heading. Goal-specific competing links are links with cosine values equal to or greater than 80% of the value of the cosine for the “correct” link. Both competing headings and competing links are highlighted in yellow. The underlying CoLiDeS cognitive model (Kitajima, Blackmon, & Polson, 2000) predicts that users first focus their attention on the subregion that is most similar in meaning and then select the link in that subregion that is most similar in meaning. Thus, the descending values of goalheading cosines predict which heading(s) users are apt to focus on, and the descending values of goal-link cosines predict the links that users are most likely to click to find the target article. Repairing these problems makes it possible for people to find items where they are most likely to look, connecting multiple links to the target whenever required. 6 Compute predicted mean total clicks for each user goal, and test predictions in the lab This step begins with the output of the One-to-Many comparison and uses a formula for predicting the mean total clicks that users will make on the Encarta webpage. The formula is based on a multiple regression analysis of the data from three experiments. The complete multiple regression-based formula computes the predicted mean total clicks = + 1.919 + 1.420 times the number of competing headings + 0.281 times the number of competing links + 0.958 if correct link is unfamiliar + 1.231 if correct link has weak scent The minimum path is a single click on the correct link that leads to accomplishing the goal, and mean predicted total clicks for a task with no usability problems is 1.919 clicks. Figure 8 shows the sorted results of the One-to-Many comparison for the goal “Find an encyclopedia article about Fern.” This is a task that has no predicted usability problems. The correct heading is “Life Science,” and “Life Science” is the heading predicted to be the heading that users will focus on. The correct link is “Plants,” and the prediction is that “Plants will almost always be clicked first. The link “Plants” is not unfamiliar, and it is not a weak-scent link (goal-link cosine <0.10). Therefore, 1.919 is the predicted mean total clicks for Find Fern. Actual data from the lab show that these predictions are very accurate for undergraduate performance on this task: • 1.1 mean total clicks on links for a minimum one-click path (vs. 1.919 predicted) • 97% first clicked a link under Life Science, and 90% clicked Plants • 100% success rate – everyone completed task in 1 or 2 clicks • 9 seconds = mean solution time Created July, 2004 15 Figure 8. Sorted results of One-to-Many comparison for the goal “Find encyclopedia article about Fern,” with the “correct” heading and “correct” link highlighted in green. A stunning contrast is provided by the task “Find an encyclopedia article about Hmong.” Figure 9 shows the results of the One-to-Many comparisons for the Hmong task. The “correct” heading is Social Science, but the cosines for three headings are all higher than the cosine for Social Science, making these headings goal-specific competing headings. Competing headings are serious problems, pulling the user’s attention away from the “correct” heading. The formula adds 1.42 clicks for each of the three competing headings. The “correct” link, “Anthropology” has two additional usability problems of its own. First, it is an unfamiliar link, and that adds 0.958 to the predicted mean total clicks. In addition, it has weak scent (goal-link cosine <0.10), which adds 1.231 to the predicted mean total clicks. Finally, there are competing links under the correct and competing headings. Under the heading “History,” the highest cosine is 0.37 for the link “People in United States History,” but there are two additional competing links that exceed 80% of that 0.37 cosine, “United States History,” and “History of Asia and Australasia.” Under the heading “Art, Language and Literature” the link “Language” has the highest cosine and no competitors. Finally, under the heading “Geography,” the highest cosine is for the link “U.S. States, Territories, & Regions.” If we decide that real users will actually select that link then there is only one competing link under “Geography.” If we rule out the highest-cosine link as a likely choice (false positive), we instead choose two competing links under “Geography,” the link “Regions of the World” and the link “Countries.” To be conservative we can do the computation of mean total clicks using just one competing link under the “Geography” heading, and that makes a total of five competing links. Applying the formula to the Find Hmong goal, we get + 1.919 + 3 (1.42) for competing headings + 5 (0.281) for competing links + 0.958 for an unfamiliar correct link + 1.231 for a weak-scent link. The total is 9.773, correlated with a predicted task failure rate of approximately 75%. The actual human performance for 38 undergraduates who searched for Hmong in the unrepaired condition of the experiment fell very close to predictions: Created July, 2004 16 • 9.0 mean clicks (vs. 9.773 predicted) • 74% task failure rate (inability to find the Hmong article within a time limit of 130 seconds vs. predicted 75% task failure rate) • Mean solution time of 124 seconds (close to the time limit of 130 seconds) • First-click success rate of only 3% This predicted problem was, in fact, a serious problem for most of the people who attempted to do it in the experiment. Repairing goal-specific competing headings and goal-specific competing links requires making it possible for people to find the target item by clicking any of the links that CWW analyses predict are most likely for real people to click. For the “Find encyclopedia article about Hmong” tasks, that means making it possible to reach the Hmong article by clicking any of the three most likely links under the “History” heading, the Language link under the “Art, Language and Literature” heading, and the “Regions of the World” and “Countries” links under the heading “Geography.” Making these repairs produced significantly better performance (p <.0001) for the 38 undergraduates who searched for Hmong on the repaired webpage compared to the 38 Figure 9. Sorted results of One-to-Many comparison for the goal “Find encyclopedia article about Hmong,” with the “correct” heading and “correct” link highlighted in green. Created July, 2004 17 undergraduates who searched for Hmong on the original, unrepaired webpage: • 2.1 mean total clicks (vs. 1.919 predicted mean clicks after all the problems had been repaired) • 0% task failure rate (all 38 experimental participants in the repaired condition found the target article before the 130-second time limit expired) • Mean solution time of 41 seconds • First-click success rate of 43% The Find Fern and Find Hmong tasks are especially close to predicted values, but overall the CWW predictions are very accurate. The hit rate for predicted problems approaches 100% (0% false alarms), the success rate for repairs (requires statistically significant improvement in performance when repaired compared to the original predicted problem) ranges from 85% to 100%, and preliminary data on predicted no-problems finds correct rejections of at least 63% with no serious problems among the misses. 7 References Blackmon, M. H. (to appear 2004). Cognitive Walkthrough. In W. S. Bainbridge (Ed.), Encyclopedia of Human-Computer Interaction, 2 volumes. Berkshire. Blackmon, M. H., Kitajima, M., & Polson, P.G. (2003) Repairing usability problems identified by the Cognitive Walkthrough for the Web. Chi Letters, 5: Proceedings of CHI 2003, 497–504 (ACM Press). Blackmon, M. H., Polson, P. G., Kitajima, M., & Lewis, C. (2002). Cognitive walkthrough for the Web. Chi Letters, 4: Proceedings of CHI 2002, 463–470 (ACM Press). Kintsch, W. (1998) Comprehension: A paradigm for cognition. Cambridge, U.K. & New York: Cambridge University Press. Kitajima, M., Blackmon, M. H., & Polson, P. G. (2000). A Comprehension-based model of Web navigation and its application to Web usability analysis. In S. McDonald, Y. Waern & G. Cockton (Eds.), People and Computers XIV—Usability or Else! (Proceedings of HCI 2000, University of Sunderland, St. Peter’s Campus, UK, September 5–8, pp. 357–373). Springer. Landauer, T. K. (1998). Learning and representing verbal meaning: Latent Semantic Analysis theory. Current Directions in Psychological Science, 7, 161–164. Landauer, T. K. (2002). On the computational basis of learning and cognition: Arguments from LSA. In N. Ross (Ed.), The psychology of learning and motivation, 41, 43–84. Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato’s problem: The Latent Semantic Analysis theory of acquisition, induction and representation of knowledge. Psychological Review, 104, 211– 240. Landauer, T. K., Foltz, P., & Laham, D. (1998). An introduction to Latent Semantic Analysis. Discourse Processes, 25, 259–284. Landauer, T. K, Laham, D, & Foltz, P. W., (2000). Intelligent Essay Assessor. IEEE Intelligent Systems, 15(5) 27-31. Created July, 2004 18 8 Appendix A: Elaborated Encarta headings and links Art, Language, & Literature language art artistic artists arts artist dialect languages dialects spoken painter painting painters paintings sculpture National & Regional Literature national literature regional Literature & Writing writing writers prewriting essays revising essay writer write draft narrative readers expository writes assignments autobiography literature descriptive humorous literary diaries diary compositions novelist Architecture architecture sculpture art artists artistic statues sculptures sculptor paintings painters architects renaissance style architect gothic designs painter painting revival artist portraits styles decoration arts decorative Artists artists art painters artist paintings painting painter artistic sculpture portraits sculptures painted paint portrait architecture museum museums realism renaissance decorative arts sculptor patrons paints gallery statues florence claude geometric style Language language spoken dialect languages dialects speak grammatical vocabulary grammar expressive speaking slang english Writers & Poets poets poetry poet poems verse prose poem poetic whitman lyric poe writers rhyme literature epic literary critic novelist lowell bryant imagery dickinson themes imaginative figurative ballads romanticism novels romantic ironic metaphor hopkins rhythms joyce essays edgar simile writer Decorative Arts arts art artistic artists decorative painters artist painting sculpture paintings painter sculptures architecture designs style Legends & Folklore legends tales storytellers stories storyteller tale heroes myths story National & Regional Art art artists artist paintings artistic painting painters painter sculpture sculptures museum national museums portraits architecture painted Painting, Drawing, & Graphic Arts sketches drawing drawings artist sketch sketching pictorial drafting shading draw painting drafters artists art drafter lettering painters painter graphic artistic geometric paintings portrait portraits drawn dimensions illustrations designer representations blueprint Sculpture sculpture sculptures art artists paintings artistic artist painting painters painter architecture statues painted museum portraits museums statue portrait paint renaissance arts Periods & Styles periods period styles Photography photography photographers photographer camera film cameras shutter photographs aperture photographic photographed films exposure images pictures photograph studio Life Science science sciences biology scientific geology physics life biologist physicists Plants plants plant flowering mosses People in Life Science science sciences biology scientific life geology physics Medicine medicine medicines doctor doctors prescription sick medical clinic Invertebrate Animals animals animal zoo zoos lions jellyfish backbones elephants veterinarian Fish fish trout tuna salmon fins fishing aquarium bait swims hook shark bass sharks fishermen shellfish catch fisherman swim arf perch gills dolphin nets seafood fishes swam carp Created July, 2004 19 Algae & Fungi algae chlorophyll alga fungi protist protists kelp spore multicellular fungus yeasts protozoans hyphae molds seaweed mushrooms spores chloroplasts microscopic mold protozoa mushroom phyla pigments rotting ponds parasites green decaying paramecium decompose Agriculture, Foodstuffs, & Livestock agriculture agricultural livestock farmers crop crops wheat subsistence farming cultivation harvest harvesting soybeans farm hogs yields corn farmer farms grain plows cultivate peanuts surpluses cultivated oats harvested barley export fertilizers Mammals mammals mammal whales reptiles rodents reptile porpoises Reptiles & Amphibians reptiles amphibians crocodiles lizards salamanders gills toads turtles tadpoles frogs lizard snakes vertebrates scales tails fins toad hatch frog fishes eggs Biological Principles & Concepts concepts principles concept theoretical Anatomy & Physiology anatomy physiology Environment environment environments environmental adapted adapt ecology adaptation surroundings ecologists Birds birds bird feathers beaks beak wings eagle nest nests loon fly nesting hummingbirds wing robins geese hawk flew pigeons feather gull crows eagles cranes ostrich swan hummingbird pigeon owls fluttered flying Viruses, Monerans, & Protists viruses bacterium bacteria virus protozoa bacterial protozoans mumps polio viral microbes host microorganisms pathogens protists colds infectious protist infections infected rabies pneumonia measles influenza microscopic vaccine immune pasteur immunity infection antibiotics disease villain tuberculosis History history historians historian historical History of Asia & Australasia asia asian malaysia strait afghanistan southeast People in European History history european People in United States History united states United States History united states African History african africans africa nigeria ghana congo tanzania niger ethiopia kenya guinea history sudan mali sahara tribal World History & Concepts world history Ancient History history ancient historians civilization monuments antiquity civilized History of the Americas history historians European History history european europeans Geography geography geographers boundaries World Cities, Towns, & Villages cities towns suburbs villages metropolitan urban Regions of the World world regions Rivers, Lakes, & Waterways rivers lakes streams waterways dumped sewage polluted dump phosphates dumping reservoirs tributaries erie Parks & Monuments parks park Countries countries underdeveloped industrialized nations Canadian Provinces & Cities cities suburbs metropolitan provinces canadian urban canadians quebec slums canada Created July, 2004 20 Islands islands island hawaiian coral hawaii mainland philippines honolulu rico puerto caribbean indies zealand ricans indonesia Mountain Ranges, Peaks, & Landforms mountain peaks mountains ranges rugged plateaus slopes sierra valleys mt peak everest foothills rocky ridges plateau climbers highest appalachians rockies cascade mount steep summit towering himalayas alps landforms nevada hills andes range U.S. Cities, Towns, & Villages cities towns suburbs villages metropolitan rural urban Maps & Mapmaking maps map mapmakers topographic globes mapping symbols contour atlas symbol mapped elevation Oceans & Seas oceans ocean seas salinity marine deepest Exploration & Explorers explorers explorer explored exploration voyages exploring columbus voyage 1492 sailed explorations magellan cabot expeditions christopher explore portugal cartier leif claimed expedition isabella riches ferdinand newfoundland adventurers da U.S. States, Territories, & Regions states united regions Religion & Philosophy religion religions teachings religious jesus faith christianity christ christians christian philosophy testament beliefs salvation belief god preached worship mohammed followers sin islam muslims spiritual holy prayer muhammad prayers divine pray eternal bible protestants catholic churches Theology & Practices practices theology religious Mythology mythology Religious Figures religious religion worship christ salvation churches religions church puritans theology protestants saints puritan bible jesus catholic prayer beliefs protestant catholics faith teachings ritual preaching reformation christian sin testament christianity christians preached spiritual congregation preach persecution god hutchinson judaism sabbath orthodox holy prayers rituals pray sins Philosophy philosophy philosophical metaphysics plato philosophers philosopher aristotle socrates thinkers principles disciplines contemplation theology Religions & Religious Groups groups religious religion religions group protestants christ christians jesus protestant beliefs catholics teachings christianity christian worship Scripture sin theology The Occult Physical Science & Technology science physical sciences physics scientific biology geology technology Construction & Engineering construction architectural engineering engineer exterior architect constructed specifications architects masonry engineers plumbing design building foundation construct tile projects designers builders Chemistry chemistry chemist chemists biology physics science sciences chemical Earth Science earth science geology scientific sciences Computer Science & Electronics computer computers microcomputer software computerized computing video graphics electronic diskette capability science calculations electronics Machines & Tools machines machine tools wedge tool levers fulcrum lever portable Created July, 2004 21 People in Physical Science physical science sciences physics scientific Astronomy & Space Science space science astronomy shuttle sciences physicists physics biology scientific astronaut astronauts Paleontology Industry, Mining, & Fuels coal industry mining mines industries mined fuels mills industrial petroleum Physics physics physicists science sciences physicist biology geology chemistry einstein scientific mathematics Transportation transportation railroads freight rail highways railroad Communications communications networks communication telegraph broadcasting telephone communicating sender stations messages message network receiver radio television broadcast media Mathematics mathematics mathematical physics algebra science math Military Technology military technology technologies Time, Weights, & Measures measures weights measure measuring measurement metric measurements measured weight kilogram grams si weighing meter Military military armed defense strategic generals civilian Social Science social sciences science sociology sociologists scientific anthropology biology Economics & Business business businesses proprietorship entrepreneurs profit Organizations organizations organization organizational membership organized association associations Institutions institutions institutional institution ideology Political Science political science politics sciences scientific elites Psychology psychology psychologists psychologist sociology behavior anthropology scientifically psychological Law law laws enforced legal enforcement enforce tort statute wrongs Education education educational schools schooling vocational educators educating colleges curriculum certification educate universities guidance elementary enrolled training Anthropology anthropology anthropologist sociology anthropologists disciplines humankind sociologists sciences mead societies human beings culture scientifically psychology cultures Military military armed defense strategic generals civilian Sociology & Social Reform social sociology sociologists sociological sociologist needy Calendar, Holidays, & Festivals calendar holidays holiday celebrate festival celebrated celebration Archaeology prehistoric anthropologists archaeologists anthropology anthropologist Sports, Hobbies, & Pets sports athletes athlete athletic athletics sport coaches games teams basketball football hockey olympic team soccer championship tennis boxing players Sports sports athletes athlete athletic athletics sport coaches football basketball games teams hockey team olympic championship soccer coach tennis players boxing player fans champion contests sporting baseball indoor league Created July, 2004 22 Sports Figures sports athletes athlete athletic athletics sport coaches football hockey basketball games teams olympic team coach tennis championship boxing soccer players fans contests Games, Hobbies, & Recreation games sports sport athletes hockey basketball golf game soccer tennis football teams players athlete playing athletic coaches played team athletics recreation player baseball contests olympic fans championship Pets pets pet cats dogs dog cat puppy kittens Performing Arts arts art artistic artists painters sculpture sculptures architecture Theater theater actors actor playwright playwrights drama plays theatre comedy shakespeare costumes performers broadway audiences scenery macbeth tragedy scenes theaters performances stage staged elizabethan script theatres scene audience costume performer performed movie chorus Musicians & Composers composers musicians jazz music singers beethoven musician orchestra musical melody piano tunes symphony concert songs blues opera chord dance song rhythmic sing rhythm concerts dancing singing tune singer violin dancer guitar rhythms performances danced instruments sung dancers ballet sang tones instrument band drums chorus dances chant folk choir instrumental strings Cinema, Television, & Broadcasting television viewers broadcast broadcasting tv commercials radio stations entertainment media movies newspapers network news magazines advertisers networks radios communications studio appearances channels Music music jazz singers musicians songs composers tunes musical melody piano orchestra beethoven musician concert song blues symphony sing opera rhythm singing tune dance rhythmic singer concerts guitar sang dancing rhythms drums sung danced chord violin choir band chant dancer performances dancers ballet instruments dances folk chorus flute tones strings instrument harp lessons sings Dance dance dancing ballet dancer dancers dances music danced jazz musicians piano composers concert orchestra singers musician opera tunes concerts performances beethoven songs melody blues musical performers sing symphony costume Musical Instruments instruments musical instrument music jazz melody piano musicians musician guitar strings violin orchestra songs concert rhythmic tune rhythm blues song rhythms sing tones drums opera band singing trumpet singer harp sung National & Regional Literature national literature regional Literature & Writing writing writers prewriting essays revising essay writer write draft narrative readers expository writes assignments autobiography literature descriptive humorous literary diaries diary compositions novelist Architecture architecture sculpture art artists artistic statues sculptures sculptor paintings painters architects renaissance style architect gothic designs painter painting revival artist portraits styles decoration arts decorative Created July, 2004 23 Artists artists art painters artist paintings painting painter artistic sculpture portraits sculptures painted paint portrait architecture museum museums realism renaissance decorative arts sculptor patrons paints gallery statues florence claude geometric style Language language spoken dialect languages dialects speak grammatical vocabulary grammar expressive speaking slang english Writers & Poets poets poetry poet poems verse prose poem poetic whitman lyric poe writers rhyme literature epic literary critic novelist lowell bryant imagery dickinson themes imaginative figurative ballads romanticism novels romantic ironic metaphor hopkins rhythms joyce essays edgar simile writer Decorative Arts arts art artistic artists decorative painters artist painting sculpture paintings painter sculptures architecture designs style Legends & Folklore legends tales storytellers stories storyteller tale heroes myths story National & Regional Art art artists artist paintings artistic painting painters painter sculpture sculptures museum national museums portraits architecture painted Painting, Drawing, & Graphic Arts sketches drawing drawings artist sketch sketching pictorial drafting shading draw painting drafters artists art drafter lettering painters painter graphic artistic geometric paintings portrait portraits drawn dimensions illustrations designer representations blueprint Sculpture sculpture sculptures art artists paintings artistic artist painting painters painter architecture statues painted museum portraits museums statue portrait paint renaissance arts Periods & Styles periods period styles Photography photography photographers photographer camera film cameras shutter photographs aperture photographic photographed films exposure images pictures photograph studio Plants plants plant flowering mosses People in Life Science science sciences biology scientific life geology physics Medicine medicine medicines doctor doctors prescription sick medical clinic Invertebrate Animals animals animal zoo zoos lions jellyfish backbones elephants veterinarian Created July, 2004 24 Fish fish trout tuna salmon fins fishing aquarium bait swims hook shark bass sharks fishermen shellfish catch fisherman swim arf perch gills dolphin nets seafood fishes swam carp Algae & Fungi algae chlorophyll alga fungi protist protists kelp spore multicellular fungus yeasts protozoans hyphae molds seaweed mushrooms spores chloroplasts microscopic mold protozoa mushroom phyla pigments rotting ponds parasites green decaying paramecium decompose Agriculture, Foodstuffs, & Livestock agriculture agricultural livestock farmers crop crops wheat subsistence farming cultivation harvest harvesting soybeans farm hogs yields corn farmer farms grain plows cultivate peanuts surpluses cultivated oats harvested barley export fertilizers Mammals mammals mammal whales reptiles rodents reptile porpoises Reptiles & Amphibians reptiles amphibians crocodiles lizards salamanders gills toads turtles tadpoles frogs lizard snakes vertebrates scales tails fins toad hatch frog fishes eggs Biological Principles & Concepts concepts principles concept theoretical Anatomy & Physiology anatomy physiology Environment environment environments environmental adapted adapt ecology adaptation surroundings ecologists Birds birds bird feathers beaks beak wings eagle nest nests loon fly nesting hummingbirds wing robins geese hawk flew pigeons feather gull crows eagles cranes ostrich swan hummingbird pigeon owls fluttered flying Viruses, Monerans, & Protists viruses bacterium bacteria virus protozoa bacterial protozoans mumps polio viral microbes host microorganisms pathogens protists colds infectious protist infections infected rabies pneumonia measles influenza microscopic vaccine immune pasteur immunity infection antibiotics disease villain tuberculosis History of Asia & Australasia asia asian malaysia strait afghanistan southeast People in European History history european People in United States History united states United States History united states African History african africans africa nigeria ghana congo tanzania niger ethiopia kenya guinea history sudan mali sahara tribal Created July, 2004 25 World History & Concepts world history Ancient History history ancient historians civilization monuments antiquity civilized History of the Americas history historians European History history european europeans World Cities, Towns, & Villages cities towns suburbs villages metropolitan urban Regions of the World world regions Rivers, Lakes, & Waterways rivers lakes streams waterways dumped sewage polluted dump phosphates dumping reservoirs tributaries erie Parks & Monuments parks park Countries countries underdeveloped industrialized nations Canadian Provinces & Cities cities suburbs metropolitan provinces canadian urban canadians quebec slums canada Islands islands island hawaiian coral hawaii mainland philippines honolulu rico puerto caribbean indies zealand ricans indonesia Mountain Ranges, Peaks, & Landforms mountain peaks mountains ranges rugged plateaus slopes sierra valleys mt peak everest foothills rocky ridges plateau climbers highest appalachians rockies cascade mount steep summit towering himalayas alps landforms nevada hills andes range U.S. Cities, Towns, & Villages cities towns suburbs villages metropolitan rural urban Maps & Mapmaking maps map mapmakers topographic globes mapping symbols contour atlas symbol mapped elevation Oceans & Seas oceans ocean seas salinity marine deepest Exploration & Explorers explorers explorer explored exploration voyages exploring columbus voyage 1492 sailed explorations magellan cabot expeditions christopher explore portugal cartier leif claimed expedition isabella riches ferdinand newfoundland adventurers da U.S. States, Territories, & Regions states united regions Theology & Practices practices theology religious Created July, 2004 26 Mythology mythology Religious Figures religious religion worship christ salvation churches religions church puritans theology protestants saints puritan bible jesus catholic prayer beliefs protestant catholics faith teachings ritual preaching reformation christian sin testament christianity christians preached spiritual congregation preach persecution god hutchinson judaism sabbath orthodox holy prayers rituals pray sins Philosophy philosophy philosophical metaphysics plato philosophers philosopher aristotle socrates thinkers principles disciplines contemplation theology Religions & Religious Groups groups religious religion religions group protestants christ christians jesus protestant beliefs catholics teachings christianity christian worship Scripture sin theology The Occult Construction & Engineering construction architectural engineering engineer exterior architect constructed specifications architects masonry engineers plumbing design building foundation construct tile projects designers builders Chemistry chemistry chemist chemists biology physics science sciences chemical Earth Science earth science geology scientific sciences Computer Science & Electronics computer computers microcomputer software computerized computing video graphics electronic diskette capability science calculations electronics Machines & Tools machines machine tools wedge tool levers fulcrum lever portable People in Physical Science physical science sciences physics scientific Astronomy & Space Science space science astronomy shuttle sciences physicists physics biology scientific astronaut astronauts Paleontology Industry, Mining, & Fuels coal industry mining mines industries mined fuels mills industrial petroleum Created July, 2004 27 Physics physics physicists science sciences physicist biology geology chemistry einstein scientific mathematics Transportation transportation railroads freight rail highways railroad Communications communications networks communication telegraph broadcasting telephone communicating sender stations messages message network receiver radio television broadcast media Mathematics mathematics mathematical physics algebra science math Military Technology military technology technologies Time, Weights, & Measures measures weights measure measuring measurement metric measurements measured weight kilogram grams si weighing meter Military military armed defense strategic generals civilian Economics & Business business businesses proprietorship entrepreneurs profit Organizations organizations organization organizational membership organized association associations Institutions institutions institutional institution ideology Political Science political science politics sciences scientific elites Psychology psychology psychologists psychologist sociology behavior anthropology scientifically psychological Law law laws enforced legal enforcement enforce tort statute wrongs Education education educational schools schooling vocational educators educating colleges curriculum certification educate universities guidance elementary enrolled training Anthropology anthropology anthropologist sociology anthropologists disciplines humankind sociologists sciences mead societies human beings culture scientifically psychology cultures Military military armed defense strategic generals civilian Sociology & Social Reform social sociology sociologists sociological sociologist needy Calendar, Holidays, & Festivals calendar holidays holiday celebrate festival celebrated celebration Created July, 2004 28 Archaeology prehistoric anthropologists archaeologists anthropology anthropologist Sports sports athletes athlete athletic athletics sport coaches football basketball games teams hockey team olympic championship soccer coach tennis players boxing player fans champion contests sporting baseball indoor league Sports Figures sports athletes athlete athletic athletics sport coaches football hockey basketball games teams olympic team coach tennis championship boxing soccer players fans contests Games, Hobbies, & Recreation games sports sport athletes hockey basketball golf game soccer tennis football teams players athlete playing athletic coaches played team athletics recreation player baseball contests olympic fans championship Pets pets pet cats dogs dog cat puppy kittens Theater theater actors actor playwright playwrights drama plays theatre comedy shakespeare costumes performers broadway audiences scenery macbeth tragedy scenes theaters performances stage staged elizabethan script theatres scene audience costume performer performed movie chorus Musicians & Composers composers musicians jazz music singers beethoven musician orchestra musical melody piano tunes symphony concert songs blues opera chord dance song rhythmic sing rhythm concerts dancing singing tune singer violin dancer guitar rhythms performances danced instruments sung dancers ballet sang tones instrument band drums chorus dances chant folk choir instrumental strings Cinema, Television, & Broadcasting television viewers broadcast broadcasting tv commercials radio stations entertainment media movies newspapers network news magazines advertisers networks radios communications studio appearances channels Music music jazz singers musicians songs composers tunes musical melody piano orchestra beethoven musician concert song blues symphony sing opera rhythm singing tune dance rhythmic singer concerts guitar sang dancing rhythms drums sung danced chord violin choir band chant dancer performances dancers ballet instruments dances folk chorus flute tones strings instrument harp lessons sings Dance dance dancing ballet dancer dancers dances music danced jazz musicians piano composers concert orchestra singers musician opera tunes concerts performances beethoven songs melody blues musical performers sing symphony costume Musical Instruments instruments musical instrument music jazz melody piano musicians musician guitar strings violin orchestra songs concert rhythmic tune rhythm blues song rhythms sing tones drums opera band singing trumpet singer harp sung Created July, 2004 29 9 Appendix B: Sample user goals for Online Encyclopedia 131 words 0.83 cosine with full encyclopedia article Find encyclopedia article about Audiometer Audiometer, instrument for testing hearing. The audiometer is an essentially simple instrument that produces pure tones of various fixed pitches (frequencies) heard through headphones. Hearing is tested one ear at a time. The operator can switch between frequencies and repeat the process with each frequency. Typically, sensitivity may be tested at frequencies of 125 hertz (Hz, or cycles per second), 250 Hz, 500 Hz, 1000 Hz, 2000 Hz, 4000 Hz, 8000 Hz, and 12,000 Hz. As an alternative to testing the normal mode of hearing through headphones, hearing by bone conduction can be tested. Hearing is never uniform over all frequencies and commonly varies widely at different frequencies. Internally, audiometers consist of a transistorized, variablefrequency audio oscillator—usually a simple feedback device—capable of producing a sinusoidal (near sine-wave) output. 142 words 0.84 cosine with full encyclopedia article Find encyclopedia article about Belemnite Belemnite, extinct group of marine animals that resembled squid in form and probably behavior, but possessed a unique internal shell, which was easily fossilized. The largest fossilized belemnite shells are about 2 m (about 6 ft) long and about 5 cm (2 in) wide. Belemnites were similar in shape to squid. The body consisted of a torpedo-shaped head and ten tentacles. The head contained the shell and all internal organs. As with squid, belemnites possessed a beaklike mouth for eating, and an ink sac. Belemnites evolved from a primitive ancestor with an external shell. As with squid, rapidly swimming belemnites sometimes suffered damaging collisions, as evidenced by numerous fossils of broken shells that appear to have healed. Identification of individual belemnite species in a rock layer enables paleontologists to date the rock layer. Belemnites are in the phylum Mollusca and class Cephalopoda. 109 words 0.83 cosine with full encyclopedia article Find encyclopedia article about Cybernetics Cybernetics, interdisciplinary science dealing with communication and control systems in living organisms, machines, and organizations. Cybernetics developed as the investigation of the techniques by which information is transformed into desired performance. One of the basic tenets of cybernetics is that information is statistical in nature and is measured in accordance with the laws of probability. The measure of probability is known as entropy (see Information Theory; Thermodynamics). Purposive behavior in humans or in machines requires control mechanisms that maintain order by counteracting the natural tendency toward disorganization. Cybernetics has also been applied to the study of psychology, servomechanisms, economics, neurophysiology, systems engineering, and the study of social systems. 185 words 0.88 cosine with full encyclopedia article Find encyclopedia article about Fern Created July, 2004 30 Fern, common name for any of a division of cryptogamous (spore-producing) plants. Ferns are found throughout the world. Most grow in damp, shady places, although certain species grow on dry ground, soil, or rocks. Ferns are among the oldest land plants. Tree ferns have woody trunks without branches, topped with clusters of feathery leaves, or fronds. Most ferns, however, have no trunks, and fronds grow directly from a short underground stem. The asexual, or sporophyte, generation represents the fern plant as it is commonly known. There are two major groups of ferns, leptosporangiate and eusporangiate. In leptosporangiate ferns, the sporangium develops from the outer derivative of a single epidermal cell, is slender-stalked, and produces fewer than 64 spores. In eusporangiate ferns, the sporangium develops from the inner derivatives of several epidermal cells, is sessile, or thin-stalked, and produces more than 256 spores. The prothallium, which in no way resembles the asexual fern plant, is a small, flat, heart-shaped structure with a number of rhizoids growing on its underside. Ferns used as pot plants are usually tropical species. Scientific classification: Ferns make up the division Filicinophyta. 155 words 0.92 cosine with full encyclopedia article Find encyclopedia article about Ferreting Ferreting, use of ferrets to flush rats or rabbits from underground channels and paths. It is believed that the domestic ferret was derived from the European polecat, Mustela putorius, as an aid in hunting rabbits. From North Africa, the Romans presumably carried the ferret and the pastime of ferreting to Europe and elsewhere. Female ferrets, because they are smaller than males and able to pursue their prey through smaller passageways, are preferred for ferreting. Ferrets are still used in Europe for reducing rodent populations on farms and in England for hunting rats along hedgerows. In ferreting for rabbits, the ferret is sometimes muzzled to prevent it from killing a trapped rabbit underground. Ferrets are able to work at seven weeks old. They hunt instinctively and without training, but it has long been a custom in many areas to cross breed ferrets with wild polecats on the assumption that such crosses produce more vigorous ferrets. 103 words 0.90 cosine with full encyclopedia article Find encyclopedia article about Frequency Modulation Frequency Modulation (FM), system of radio transmission in which the carrier wave is modulated so that its frequency varies with the audio signal being transmitted. FM broadcasting stations can be operated in the very-high-frequency bands at which AM interference is frequently severe; commercial FM radio stations are assigned frequencies between 88 and 108 Mhz. In 2000, there were about 5,770 FM stations. In 1961 the Federal Communications Commission authorized FM stereophonic broadcasting. Thereafter, the FM band drew increasing numbers of listeners to popular as well as classical music, and commercial FM stations began to draw higher audience ratings than AM stations. 205 words 0.82 cosine with full encyclopedia article Find encyclopedia article about Hmong Hmong, minority ethnic group that lives primarily in China and Southeast Asia. About 2 million Hmong live in Southeast Asian countries, such as Vietnam, Laos, Thailand, and Myanmar. Created July, 2004 31 Another 10 million Hmong live in the southern provinces of China. The United States has the largest Hmong refugee community, with a population of about 300,000 in 2001. The word Hmong, which means “man” in the Hmong language, is the name used by the Hmong people themselves. During the Vietnam War, some Hmong began translating the name Hmong as “free man” to express their desire for political independence. The Hmong language contains seven tones. Within Hmong society, subgroups speak slightly different versions of the Hmong language. The largest subgroups are White Hmong, Red Hmong, Blue or Green Hmong, and Striped Hmong. A Hmong bride joins the clan of her husband. With French encouragement, many Hmong turned to opium cultivation during World War II (1939-1945). Hmong in the United States. Between 1975 and 1994, more than 110,000 Hmong refugees resettled in the United States. Because Hmong tend to have large families, these communities have grown rapidly. Hmong families have faced considerable challenges in adapting to American life. Hmong women have earned money selling their colorful needlework. Created July, 2004