Information Retrieval List Digest 186 (November 5, 1993) URL = http://hegel.lib.ncsu.edu/stacks/serials/irld/irld-186 IRLIST Digest ISSN 1064-6965 November 5, 1993 Volume X, Number 42 Issue 186 ********************************************************** I. NOTICES A. Meeting Announcements/Calls for Papers 1. British Computer Society IR '93/94 Colloquium 2. Microneuro '94 C. Miscellaneous 1. Report on TREC-2 Conference ********************************************************** I. NOTICES I.A.1. Fr: Ruben Leon Re: BCS IR 93/94 Colloquium: Call for Papers THE BRITISH COMPUTER SOCIETY INFORMATION RETRIEVAL SPECIALIST GROUP 16th Annual Colloquium 22-23 March 1994 Drymen (by Loch Lomond), Scotland Hosted by the University of Strathclyde Department of Information Science CALL FOR PAPERS: The 16th BCS Colloquium on IR will be held on 22nd P 23rd March 1994 in Buchanan Highland Hotel, Drymen. Drymen is an atmospheric village located some 15 miles north of Glasgow and 30 minutes walk from Loch Lomond. The colloquium will be hosted by the Dept. of Information Science of Strathclyde University. The Colloquium is aimed at researchers who want to offer their work to the IR community in an informal atmosphere. The Conference will follow the established informal and convivial style of conference for which the BCS IR Group is known. Papers are invited on any topic in the field of computerised information retrieval. Possible themes could include: Hypermedia Retrieval of images Distributed network systems Language and communication protocols for distributed systems Natural Language Processing Logic and information retrieval Interface issues in information retrieval Information retrieval in library systems Parallel distributed models for information retrieval Legal aspects of computerised information retrieval Evaluation and testing of information retrieval systems SUBMISSIONS: Papers should be submitted in the first instance in paper or electronic form (or both), no later than 6 February 1994. A final submission of accepted papers should reach the Conference Organiser no later than the start of conference, and should include two camera ready copies of each paper together with an electronic version on a 3 1/2S Macintosh or PC formatted floppy, in Microsoft Word, ASCII, or LATEX format. Conference proceedings will be typeset in a Macintosh with Microsoft Word. Submissions should be addressed to: Ruben Leon, IR Conference Organiser, Dept. of Information Science, University of Strathclyde. Glasgow G1 1XH, Scotland. E-mail: Conference@dis.strath.ac.uk. All enquiries and conference bookings should be addressed to: IR Conference Organiser, Department of Information Science University of Strathclyde Glasgow G1 1XH, Scotland. E-mail: Conference@dis.strath.ac.uk ********** I.A.2. Fr: Leonardo Reyneri Re: Microneuro '94 MICRONEURO 94 The Fourth International Conference on Microelectronics for Neural Networks and Fuzzy Systems Torino (I), September 26-28, 1994 FIRST CALL FOR PAPERS This conference is the fourth in a series of international conferences dedicated to all aspects of hardware implementations of Neural Networks and Fuzzy Systems. MICRONEURO has emerged as the only international forum devoted specifically to all hardware implementation aspects, giving particular weight to those interdisciplinary issues which affect the design of Neural and Fuzzy hardware directly. TOPICS: The conference program will focus upon all aspects of hardware implementations of Neural Networks and Fuzzy Systems and their applications in the real world. Topics will concentrate upon the following fields: - Analog and mixed-mode implementations - Digital implementations - Optical systems - Pulse-Stream computation - Weightless Neural systems - Neural and Fuzzy hardware systems - Interfaces with external world - Applications of dedicated hardware - VLSI-friendly Neural algorithms - New technologies for Neural and Fuzzy Systems Selection criteria will be based also on technical relevance, novelty of the approach and on availability of performance measurements for the system/device. INFORMATION FOR AUTHORS: All submitted material (written in English) will be refereed and should be typed on A4 paper, 1-1/2 spaced, 12 point font, 160x220 mm text size. All accepted material will appear in the proceedings. PAPERS should not exceed 10 pages including figures and text. Also reports on EARLY INNOVATIVE IDEAS will be considered for presentation. In this case the submission should be a short description of the novel idea, not exceeding 6 pages in length, and it must be clearly marked ``Innovative Idea''. The most interesting papers and ideas will be published in a special issue of IEEE MICRO. SUBMISSIONS: Six copies of final manuscripts, written according to the above requirements, shall be submitted to the Program Chairman. Submissions arriving late or significantly departing from length guidelines, or papers published elsewhere will be returned without review. Electronic versions of the submission (possibly in LATEX format) are kindly welcome. DEADLINES Submission of paper and/or ideas May 30, 1994 Notification of acceptance July 15, 1994 THE WORKSHOP VENUE: The venue of MICRONEURO '94 is Torino, the historic and beautiful center of Piemonte. The town is surrounded by the highest mountains in Europe and by beautiful hills and landscapes. The region is also famous for its excellent wines. MICRONEURO '94 will be held at the Politecnico di Torino. The venue is conveniently located close to the town centre, with many restaurants and cafes close by. General Chair: H.P. Graf AT T Bell Laboratories Room 4 G 320 HOLMDEL, NJ 07733 - USA Tel. +1 908 949 0183 Fax. +1 908 949 7722 Program Chair: L.M. Reyneri Dip. Ingegneria Informazione Universita' di Pisa Via Diotisalvi, 2 56126 PISA - ITALY Tel. +39 50 568 511 Fax. +39 50 568 522 E.mail lmr@pimac2.iet.unipi.it Organisation: COREP Segr. MICRONEURO '94 C.so Duca d. Abruzzi, 24 10129 TORINO - ITALY Tel. +39 11 564 5108 Fax. +39 11 564 5199 ********** I.B.2. Fr: asmeaton@compapp.dcu.ie Re: Report on TREC-2 Conference Report on TREC-2 (Text REtrieval Conference) 30 August - 2 September, Gaithersburg, USA INTRODUCTION: As part of an effort to encourage research into text retrieval from large and diverse document collections, the first Text REtrieval Conference (TREC-1) was held in Gaithersburg, Md., in 1992. This forum provided researchers with a large collection of textual materials, queries and associated relevance judgements, and a uniform scoring procedure. The conference, co-sponsored by the U.S. Advanced Research Projects Agency (ARPA) and the U.S. National Institute for Standards and Technology (NIST), was a benchmarking exercise which involved gauging the relative effectivenesses of many different approaches to the indexing and retrieval of large volumes of text. A second conference/workshop (TREC-2) was held in early September 1993 and was the culmination of the experimental runs carried out at over 31 sites where information retrieval research is carried out across the world. The 19 full TREC-2 participants and TIPSTER groups with their respective approaches to information retrieval were (U.S. unless stated): BELLCORE - SMART document preprocessing and Latent Semantic Indexing; CARNEGIE MELLON UNIVERSITY - NLP-based indexing; CITRI, Royal Melbourne Institute of Technology (Australia) - document structure and efficiency issues; CITY UNIVERSITY, LONDON (UK) - variant of probabilistic model and probabilistic weighting functions; CORNELL UNIVERSITY - Vector Space Model/SMART system; ENVIRONMENT RESEARCH INSTITUTE OF MICHIGAN - n-gram indexing/retrieval; GE RESEARCH AND DEVELOPMENT CENTER - building complex boolean queries; HNC - TIPSTER group learning a reduced dimensionality index space; INSTITUTE FOR DECISION SYSTEM RESEARCH - Bayesian networks; NEW YORK UNIVERSITY - NLP-based indexing by word pairs; QUEENS COLLEGE (CUNY) - variant of probabilistic model using PIRCS system and spreading activation; RUTGERS UNIVERSITY - combination of results of different retrieval strategies; SIEMENS CORPORATE RESEARCH INC. - SMART retrieval & query expansion using WordNet; SWISS FEDERAL INSTITUTE OF TECHNOLOGY (ETH) (Switzerland) - efficient implementation of RSV metric; THINKING MACHINES CORPORATION - various vector space model experiments, concerned with efficiency of execution; TRW SYSTEMS DEVELOPMENT DIVISION- hardware filtering; UNIVERSITAET DORTMUND (GERMANY) & CORNELL UNIVERSITY - variant of probabilistic model with learning of parameter values; UNIVERSITY OF CALIFORNIA, BERKELEY - variant of probabilistic model with logistic regression for probability estimates; UNIVERSITY OF MASSACHUSETTS - TIPSTER group using a Bayesian Inference Network approach; VERITY, INC. - machine learning for TOPIC IR system; VPI&SU - combining results of multiple searches. LOGISTICS: As with the first TREC, participants in TREC-2 worked with approximately one million documents (2 gigabytes of text data), retrieving lists of documents that could be considered relevant to each of 50 topics in what was called "ad hoc" querying. A second information retrieval paradigm used was where 50 retrieval topics were known in advance and new documents were to be matched against the 50 standard queries simulating a "routing" operation. In both cases the queries were not really queries at all but carefully honed user need statements and were thus extensive descriptions of the topic of interest. Participating groups were allowed to do completely automatic query construction, manual query formulation or to simulate relevance feedback. The test data used consisted of newspaper stories (Wall Street Journal and San Jose Mercury News), Associated Press Newswire articles, U.S. patent applications and articles from the Federal Register, the Ziff database and the U.S. Department of the Environment, all in all, a deliberately heterogeneous mix of document types and document lengths. The test data was distributed by NIST and was installed by the participants at their research sites, in addition to some test topics and relevance assessments. Participating groups fine-tuned their retrieval strategies and were then sent the new topics for ad-hoc querying and 1 gigabyte of new test data for the pre-defined routing queries. The ranking results from each site were then sent back to NIST who pooled together the rankings and had teams of assessors manually evaluate the relevance of each document appearing in the top 100 documents from at least one site, for each of the 50 ad hoc and 50 routing queries. A total of 41 different ad hoc runs (from 25 groups) and 40 different routing runs (from 23 groups) were pooled to generate the set for manual relevance assessment. As was expected, different systems retrieved different sets of documents in their top rankings but there was a much higher overlap in retrieved document sets as compared with the first TREC, possibly because the systems in TREC-2 were better. Each participant in TREC-2 set their own baseline effectiveness levels using the trial queries and relevance assessments provided at the start of TREC-2 and then improved upon, or deteriorated their relative effectivenesses on the official runs. No relevance assessments were available at the time the official runs were being completed and there was little time given in which to complete these runs, so there wasn't much tinkering that could be done in the time allowed. This ensured that no system had an unfair advantage over any others. EVALUATION: The issue of evaluation has always been one of debate in information retrieval and within TREC there is scope for even more discussion than normal. For the "official" results submitted by each group, NIST calculated a range of statistical performance figures including averaged recall-precision, recall-fallout, and precision figures at 5,10,15,20,30,100,200,500 and 1000 documents. A major improvement in the evaluation of TREC-2 over the first TREC is the fact that the top 1000 ranking and not just top 200 documents per topic were submitted by the groups. The way in which evaluation figures were calculated and averaged were also improved upon. There is a real problem with using the standard measures for information retrieval evaluation on something like TREC; looking at averages of averages is very superficial and hides most of what is actually going on with respect to performance. In TREC-2 it was possible to do some failure analysis on the data before the workshop and this showed some interesting features like the fact that "long" documents were being retrieved by most approaches but were not proving relevant and that systems which yielded poor levels of precision averaged over 50 topics actually did well, often best, for some of those 50 topics ! In fact there were 21 groups for which there was at least one topic on which their system had the best average precision. The message to be found here is that there is much work to be done on the data generated by different retrieval approaches to try and explain some of the results. THE WORKSHOP: The TREC-2 workshop in September 1993 was open only to participating systems and government sponsors and was even more open and sharing and workshop-like than TREC-1. Each participant presented an overview of their system and the performance as measured using the evaluation methods outlined earlier were available for all official runs for all systems. There are many different approaches to information retrieval represented among the TREC-2 participants grouped roughly into probabilistic models and variants thereof, vector space approaches, NLP- based, bayesian networks, query expansion and dimensionality reduction, boolean query construction, combination of results of different retrieval strategies, explorations into document structuring, ... as well as some outliers like retrieval using n-grams, word pairs, hardware approaches, and some work on efficiency issues. Generalising results across systems and across approaches is difficult but some trends have already emerged. Simple systems which do simple things are still doing really well and the more complex ones are catching up and in some cases surpassing the simple approaches. This result was expected after the first TREC where simple systems did well and more complex ones generally did not. Term weighting has also emerged as something which counts as especially important. There is also a large spread of levels of effectiveness among systems. An irritating aspect of the way information retrieval evaluation using the standard measures is that this does not show the full power of the systems; to say that system retrieves 13 relevant documents in its top 20 ranked set from a collection of 1,000,000 means that that system really is a good computational tool to have, but recall and precision values hide this fact. The inclusion of recall-fallout tables addresses this somewhat but these still belie the fact that the information retrieval techniques available are really quite good. The overall measurements in TREC-2 show an improvement in effectiveness over the first TREC and whereas some of this is due to the fact that ranking was done to top 1000 and not to top 200, it may also be due to the systems being better. This could be because systems in TREC-2 are more fine tuned than before as most TREC-2 participants were also in TREC-1 and they would have thus been able to anticipate, if not always fully overcome, the engineering problems entailed when wrestling with 2 or 3 Gbytes of text and the associated indexes, etc. TREC-2 seemed to have less problems with engineering the volume of the data than before, probably because most groups had been through it before. >From the outset, efficiency issues were never foremost in TREC which is benchmarking retrieval effectiveness and is not directly concerned with the engineering aspects of large IR systems. Some of the figures for indexing and retrieval operations show a very large range of equipment and performance times, varying from retrieval from 2 gbytes in less than 5 seconds where the entire inverted file is held in main memory, to retrieval for a single query measured in hours of CPU and elapsed time implemented on a PC which decompresses and scans the text as it is being read from the CD-ROM. The message here is that you don't need massive computing resources to take part in TREC ... it helps and it makes things easier, but it is not mandatory. In fact, as described in the accompanying call for TREC-3, there is a category of participation within TREC for computationally intensive approaches which allows a group to use only a subset of the entire collection. Finally, TREC-2 did not have as much work on sub-document retrieval as was expected. This may be due to the fact that relevance judgements are dichotomous and do not indicate which PART of a document is relevant, as in most IR test collections. This is something which is being looked at in the next TREC. WIND UP: At the start of this report we cautioned about making comparisons between systems and approaches which took part in TREC because of the number of variables involved. This then begs the question of why bother doing TREC if the results of different systems cannot be compared ? The answer lies in the objective of the TREC initiative, which were defined by Donna Harman as 1. to increase research in information retrieval carried out on large-scale test collections 2. to provide a forum for communication among academic, industrial and other interested parties 3. to foster the transfer of technology between research laboratories and commercial products 4. to present a state of the art showcase of retrieval methods Certainly the first and last of these goals have been achieved; the second goal looks like having been accomplished and as for the third, only time will tell. Direct comparisons between systems and approaches taken in TREC are extremely dodgy and only broad stroke statements about effectiveness as made in this report, can be made. For information on TREC-3, contact Alan Smeaton at asmeaton@commpapp.dcu.ie. ********************************************************** IRLIST Digest is distributed from the University of California, Division of Library Automation, 300 Lakeside Drive, Oakland, CA. 94612-3550. Send subscription requests to: LISTSERV@UCCVMA.BITNET Send submissions to IRLIST to: IR-L@UCCVMA.BITNET Editorial Staff: Clifford Lynch calur@uccmvsa.ucop.edu or calur@uccmvsa.bitnet Nancy Gusack ncgur@uccmvsa.bitnet or ncgur@uccmvsa.ucop.edu Mary Engle meeur@uccmvsa.bitnet The IRLIST Archives is now set up for anonymous FTP, as well as via the LISTSERV. Using anonymous FTP via the host dla.ucop.edu, the files will be found in the directory pub/irl, stored in subdirectories by year (e.g., /pub/irl/1993). Using LISTSERV, send the message INDEX IR-L to LISTSERV@UCCVMA.BITNET. To get a specific issue listed in the Index, send the message GET IR-L LOGYYMM, where YY is the year and MM is the numeric month in which the issue was mailed, to LISTSERV@UCCVMA (Bitnet) or LISTSERV@UCCVMA.UCOP.EDU. You will receive the issues for the entire month you have requested. These files are not to be sold or used for commercial purposes. Contact Nancy Gusack or Mary Engle for more information on IRLIST. THE OPINIONS EXPRESSED IN IRLIST DO NOT REPRESENT THOSE OF THE EDITORS OR THE UNIVERSITY OF CALIFORNIA. AUTHORS ASSUME FULL RESPONSIBILITY FOR THE CONTENTS OF THEIR SUBMISSIONS TO IRLIST.