1. Problem Definition

Humans live in a dynamic world, where actions bring about consequences, and the facts and properties associated with entities change over time. For this reason, temporally grounded events are the very foundation from which we reason about how the world changes. To be sure, named entity recognition is crucial to analyst reporting, information extraction, and question-answering systems; but without a robust ability to identify and extract events and time-stamps from a text, the real "aboutness" of the article can be missed. Moreover, entities and their properties change over time as well; hence a database of assertions about entities will be incomplete or incorrect if it doesn’t reflect such time-stamps (e.g., the status of the World Trade Center Buildings before and after Sept. 11, 2001). To this end, event recognition drives basic inferences from text.

The purpose of this workshop is to address the problem of how to answer temporally-based questions about the events and entities in news articles. Currently, questions such as those shown below are not generally supported by Q&A systems.

1. a. Is Gates currently CEO of Microsoft?

What characterizes these questions as beyond the scope of current systems is the following: they refer, respectively to the temporal properties of the entities being questioned, the relative ordering of events in the world, and events that are mentioned in news articles, but which have not occurred at all.

1. Workshop Goals

There has recently been a renewed interest in temporal and event-based reasoning in language and text, particularly as applied to information extraction and reasoning tasks (cf. Mani and Wilson, 2000, and 2001 ACL Workshop on Spatial and Temporal Reasoning). Several papers from the workshop point to promising directions for time representation and identification (cf. Filatova and Hovy, 2001, Schilder and Habel, 2001). Many issues relating to temporal and event identification remain unresolved, however, and it is the goal of this workshop to address these questions. Specifically, the goals of this workshop are twofold: (a) to examine how to formally distinguish events and their temporal anchoring in text (news articles); and (b) to evaluate and develop algorithms for identifying and extracting events and temporal expressions from texts.

Relative to the first goal above, there are four basic research problems that will be addressed:

1. Time stamping events (identifying an event and anchoring it in time);
2. Ordering events with respect to each other (relating more than one event in terms of precedence, overlap, and inclusion);
3. Reasoning about the ramifications of an event (what is changed by virtue of an event);
4. Reasoning about the persistence of an event (how long does an event or the outcome of an event persist).

To answer these problems, two things are needed: a specification language and an annotated Gold Standard corpus. A specification language, TenseML, will be defined and developed as a deliverable of the workshop. This XML-compliant language should formally model most of the following properties of time and events:

• How to represent the interval values of events (time-stamping);
• How to represent aspectual properties of an event (what phase of an event is being time-stamped);
• How to represent all possible temporal ordering relations between two events;
• How to model shallow (entailed) ramifications of an event (what related events are triggered by an event’s occurrence);
• How to model when a state persists and when it does not (what states follow from an event);

Once the initial definition and specification of TenseML is complete, it will be necessary to begin annotation on a large number of news articles, in order to create a Temporal Gold Standard (TIMEBANK). This would most likely entail the annotation of at least 300-500 articles, taken from either Reuters, the Factiva Media Base, or existing corpora such as TDT and TREC. TIMEBANK, the annotated corpus, will be a second deliverable of the workshop. It should be noted that no such corpus of annotation currently exists.

The specification language will suggest but not determine the nature of how answers to temporal questions are best presented to the user. This remains largely an issue of habitability and usability of the application. Nevertheless, answers to temporal questions may take one of several forms:

1. Selections from database entries, populated from the appropriate information extraction algorithms;
2. Textual fragments from news articles, indicating total or partial answers to the question;
3. Answers may be abstracted and represented visually in terms of a timeline or a hyperbolic visualization algorithm.

Phase II of the workshop addresses the second goal mentioned above. This involves the evaluation of existing, and development of new temporal extraction algorithms. The four research problems given above correspond roughly to extraction algorithms of increasing degrees of sophistication and complexity. Time stamping events is not too dissimilar from named entity recognition; event ordering identification is somewhat similar to relational parsing; and capturing persistence and ramification properties of events is similar to identifying dependencies in a dependency grammar.

The algorithms will be applied and tested against the development corpus of the gold standard, TIMEBANK. Evaluation against a blind test set will measure for accuracy of answers for a range of questions, as defined by the participants, paying particular attention to target the specific temporal properties of the text with different questions.

1. Group Composition

The core members of the team have been selected for their theoretical and applied experience in information extraction, question-answering systems, and research on temporal and event reasoning. The Team Lead, James Pustejovsky, has worked extensively on event-based semantic models for natural language, both theoretical and applied. Pustejovsky helped write the early MUC specification for time while a TIPSTER contractor, and, at his company LingoMotors, has developed a large (2500) lexical database of event types for use in information retrieval and database querying systems.

The following individuals will be core members of the workshop team:

Robert Ingria, LingoMotors Inc.

Dragomir Radev, University of Michigan

Janet Hitzeman, MITRE

John Moynes, Hapax Information Systems

Jose Castano, Brandeis University

James Allen, University of Rochester

Robert Gaizauskus, University of Sheffield

Additionally, there will be part-time involvement requested from individuals, who may contribute to a sub-theme of the workshop. They are proposed below:

Lisa Ferrro, MITRE

John Frank, MetaCarta

Chris Porter, Factiva

Antonio Sanfilippo, SRA

Len Schubert, University of Rochester

2. Project model and schedule/timeline

The 6-week workshop will be held over six months, meeting at generally one-week intervals throughout this period. The proposed schedule is given below (see attached schedule sheet as well):

1. January kickoff meeting: 1 week meeting
2. March: 3 day weekend meeting
3. April: 1 week meeting
4. May: 1 week meeting
5. June: 1 week meeting
6. July: 1 week meeting (Close up and evaluation)

The reasons for this structure are as follows. The define and design phase of the workshop is (largely) independent and antecedent to the algorithm development phase. The individual team members most needed for this phase need not participate in the development phase, Phase II. Hence, to facilitate these requirements and time constraints of potential participants, we have structured the workshop as a sequence of related one-week sessions.

1. Workshop Deliverables

The workshop hopes to provide an understanding of what the limits of the current technologies are, while providing some preliminary capabilities for answering questions involving temporal expressions. Several important artifacts will be generated as a result of the workshop:

- A standard markup language for temporal and event expressions, TenseML;

Significantly, the results of this workshop will enable the community to begin addressing an entirely new type of question-answering capability, and one that is necessary for answering questions pertaining to the deeper content of news articles.

Allen, J. "Maintaining Knowledge about Temporal Intervals", Communications of the ACM, 26(1):"832-843.

Filatova, E. and E. Hovy (2001) "Assigning Time-Stamps To Event-Clauses", in Proceedings of ACL Workshop on Temporal and Spatial Information Processing, Toulouse, France, July, 2001.

Mani, I. and G. Wilson (2000) Robust Temporal Processing of News", in Proceedings of the 38^th Annual Meeting of the ACL, Hong Kong.

Radev, D. and K. McKeown (1998) Generating Natural Language Summaries from Multiple on-line sources, Computational Linguistics, 24(3):469-500.

Schilder, F. and C. Habel (2001) "From Temporal Expressions to Temporal Information: Semantic Tagging of News Messages", in Proceedings of ACL Workshop on Temporal and Spatial Information Processing, Toulouse, France, July, 2001.

Sundheim, B., and Grishman, R., eds. 1995. Proceedings of the Sixth Message-Understanding Conference (MUC-6). San Francisco, Calif.: Morgan Kaufmann.

1. Is Gates currently CEO of Microsoft? (time-stamp question)

2. When does the seminar take place? (punctual event question)

3. How long did the hostage situation in Berlin last? (Duration of event question)

4. On what days were there bombings in the Middle East? (Quantified event question)

5. What airplane crashes occurred shortly after assassinations? (Quantified event question with relative event orderings)

6. What terrorist actions occurred within a week of political speeches by extremist governments? (Quantified event question with relative event ordering)

7. What bombings have occurred during the occupation of the West Bank? (Quantified event question with durative event overlapping)