net.sourceforge.nite.datainspection.timespan
Class SegmentBasedInspection

java.lang.Object
  net.sourceforge.nite.datainspection.timespan.SegmentBasedInspection

public class SegmentBasedInspection
extends java.lang.Object

Inspection tool for timeline segmentations (gapped or non gapped) that investigates whether two annotators identified same segments. Segments detected by two annotators are taken to be the same if they differ in start or end time at most a (configurable) threshold 'th'. Kappa and alpha are calculated by comparing the labeling for each aligned pair of segments.

Segment alignment based inspection

This section describes an inspection tool that looks at full segments rather than boundaries. Back to the summary in the nite.datainspection package documentation: Data inspection consists of

1. finding out whether separate annotators identified the same Items (segments, units for labeling),
2. finding out whether comparable Items have been assigned the same Values (labels, classes, categories) by the annotators and
3. finding out where disagreement lies, i.e. what Values are confused with each other in what situations; what type of Items are most often NOT identified by both annotators at the same time; etc.
4. (Investigating the nature of the errors that annotators made, and deciding how important these errors are, given the use for which the annotations were created.)

The Segment Based inspections are involved with another approach to step one: finding out whether separate annotators identified the same items or events in the data. In this analysis, the focus is on detection whether two annotators identified the same segments at the same moment. The image below shows a (hypothetical) example annotation fragment. The key point of the inspections described in this section is that the pair of blue annotation elements as well as the pair of red ones are both considered to be "the same Item identified by both annotators, and given the same label", with the precision of the timing slightly off. This means that a confusion table of this annotation would show two instances of confusion between the labels 'green' and 'brownish-green', and no confusion between red and green or blue and green, even if there is some overlap in time for these labels. Furthermore this analysis does not take the length of the aligned segments into account: a (dis)agreement between two segments counts equally for two long or two short segments.

an example 'timespan' annotation for two annotators

A usual position that people take is something like "In order to align the two annotations, it was decided that two segments referred to the same gesture if they covered the same time span, plus or minus a quarter of a second at the onset or end of the gesture" [MUMIN], with varying thresholds of allowed deviation (e.g. MUMIN: 0.25 sec.; Natasa|gaze: 0.8 sec.)

Precision

Varying threshold 'th' gives information about the precision with which two annotators identified the same segments. Note however that 'th' should be low enough compared to the segment lengths in the annotations.

         ========
         ===Aligning f vs e
         ---
         - th 0.0
         9 aligned segments 
          For ann1: 2%.
          For ann2: 3%.
         ---
         - th 1.5
         242 aligned segments 
          For ann1: 74%.
          For ann2: 90%.
         ---
         - th 3.0
         248 aligned segments 
          For ann1: 76%.
          For ann2: 92%.

The list above gives an example output obtained from the SegmentBasedInspection tool, for the FOA layer of the AMI corpus, with several variations of 'th'. Again, we look at the visualisations of the alignments, because of course we are afraid of running into the same 'th' sensitivity that we had with the boundary alignments. The following image show an alignment produced using the tool SegmentBasedInspection. Actually (but for now you'll have to take my word for it...) it turns out that this alignment is for this annotation a lot better when it operates on segments instead of boundaries (see the boundary based inspection for example alignments). Furthermore it is less sensitive to too high threshold values. In this annotation even a treshold of 3 seconds leads to almost only correct alignments (compare to the maximum adequate threshold of about 0.3 seconds for the boundary alignment).

automatic alignment for a good threshold.

AND NOW SOME REMARKS ABOUT REASONS TO EXTEND THE ALIGNMENT ALGORITHM!

Why? Why do I feel that the alignment should be made more complicated, when the example above showed that it works quite allright?

Remember why you do reliability analysis. The reason to do reliability analysis is not only to reach a conclusion like "the alpha reliability of my annotation is 0.8 so it is good enough". The main reason to do this data inspection is to figure out the strong points and the weak points of your annotations. Hopefully identifying those strong and weak points helps you improve the annotation protocol, or decided what you can and cannot do with the data resulting from your annotation protocol.

Taking labels into consideration.

Take a look at the image below, which shows a fragment of two aligned annotations for hand gestures in the AMI corpus. The red fragments are 'no_comm_hand'. The other colours all stand for some communicative hand gesture. Note especially the alignment between the brownish-green fragment (point_p_id_RH) of the first annotator with the red fragment directly below it, next to the brown exclamation mark. The alignment procedure obviously aligned the element with the closest similarily sized element of the other annotator. When we now proceed to analyse the labelings of the aligned segments, this alignment will contribute a 'confusion between point_p_id_RH and no_comm_hand'. What we actually wanted to learn about this annotation is that the two annotators agree perfectly on the occurrence and labeling of the point_p_id_RH gesture, but disagreed more than we expected on the timing of this segment. So... it would have been nice if the alignment algorithm would have aligned the two brownish-green fragments near the exclamation mark (the are within a 'th' distance of each other...). As a second example, see the blue exclamation mark on the right: both annotators find two gestures there (and a lot of no_comm_hand), agree on the labeling of the left one and disagree on the labeling of the right one and disagree a lot on the timing of the left element. The alignment goes wrong, leading to a conclusion that there are two disagreements on labeling and one gesture found only by one annotator.

An example of an alignment that goes wrong when labels are not considered in the alignment.

The kind of alignment mistake described above is often unavoidable. When they don't happen too often, they are also not so much of a problem, since the data inspection packages aim at getting an idea of the quality of the corpus, not at getting an absolute truth about correctness of annotations. In some cases however it is quite easy to improve the alignments. In the example mentioned above, the hand gesture annotation of AMI, the no_comm_hand label is a kind of "background label" which is used whenever there is no communicative hand gesture. Since the alignment procedure aims at aligning the segments where two annotators agree on the occurrence of (in this case) a gesture, it seems sensible to leave out these background segments from the analysis. This is supported by the 'isIgnoreP' parameter in the constructor of the SegmentBasedInspection tool. If this parameter is non-null, all segments for which this predicate evaluates true are ignored in the analysis. If we run the alignment for the AMI hand gestures anew while defining "no_comm_hand" as an to-be-ignored label we get the alignment shown below.

An example of the same alignment with no_comm_hand as an ignored label.

Next step: alpha reliability on labeling of aligned segments

Supported by tool SegmentBasedInspection; some information needed. You must provide a DistanceMetric and a NOMElementToValueDelegate to the constructor of the SegmentBasedInspection tool. These are used to calculate Krippendorff Alpha on the aligned segments. These values are reported on the System.out. Furthermore confusion tables are shown on screen for every pair of annotators. For the AMI hand gesture annotation, the output is as follows (Alpha reported for all aligned hand gestures using the string type label as Value and a BooleanMetric as distance metric):

         Deriving matrices for BrigitteGreenwood vs pistek
         ==============================================================
         Alpha = 0.7226277372262774 for threshold 0.5
         ==============================================================
         Alpha = 0.8085808580858086 for threshold 1.5
         ==============================================================
         Alpha = 0.8171091445427728 for threshold 2.5

         Deriving matrices for BrigitteGreenwood vs xsobol03
         ==============================================================
         Alpha = 0.7428571428571429 for threshold 0.5
         ==============================================================
         Alpha = 0.7128712871287128 for threshold 1.5
         ==============================================================
         Alpha = 0.7256637168141593 for threshold 2.5

An example of a confusion table shown by the SegmentBasedInspection tool. Note for example the confusion concerning what hand this participant used to point... (left or both hands, point_me_LH and point_me_BH). The tool allows one to find that particular confusion on the timeline, then click on the timeline to bring up that particular stretch of video on the NXT video player.

Field Summary

java.lang.String	agentName the name of the agent for which you want to analyse the annotations.
java.lang.String	codingName the name of the Coding in which the boundaries are to be found
java.lang.String	corpusName corpus
java.lang.String	observationName observation
java.lang.String	segmentElementName the name of the Elements in the Layer in that Coding in which the boundaries are to be found
java.lang.String	segmentsLayer the name of the Layer in that Coding in which the boundaries are to be found

Constructor Summary

SegmentBasedInspection(java.lang.String c, java.lang.String o, java.lang.String codingName, java.lang.String segmentsLayer, java.lang.String segmentElementName, java.lang.String agentName, NOMElementToTextDelegate segmentToText, Predicate isIgnoreP, NOMElementToValueDelegate segmentToValue, DistanceMetric labelDistanceMetric, double thMin, double thMax, int thSteps)

Method Summary

void	collectAlignments(double thMin, double thMax, int thSteps) Collect for all threshold values the alignments for all annotator pairs, using the SegmentAligner.
void	collectClassifications() Collect for all threshold values the derived classifications for all alignments, using the SegmentAlignmentToClassificationFactory.
void	collectMatrices() Collect for all Classification pairs the confusion and coincidence matrices used for calculation of reliability and inspection of confusions.
void	collectSegments() Collect the segments for all annotators.
void	drawLegend()
void	generateThresholdValues(double thMin, double thMax, int thSteps)
Clock	getClock()
ClockFace	getClockFace() Returns the clockface of this class.
NOMWriteCorpus	getCorpus()
java.lang.String	getCorpusName()
NiteMetaData	getMetaData()
java.lang.String	getObservationName()
void	initReportPanel()
void	renderRelations()
void	renderSegments()
java.util.List	search(java.lang.String query)
void	showConfusionTables() Show the confusion tables in internal frames on the desktop

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

corpusName

public java.lang.String corpusName

corpus

observationName

public java.lang.String observationName

observation

codingName

public java.lang.String codingName

the name of the Coding in which the boundaries are to be found

segmentsLayer

public java.lang.String segmentsLayer

the name of the Layer in that Coding in which the boundaries are to be found

segmentElementName

public java.lang.String segmentElementName

the name of the Elements in the Layer in that Coding in which the boundaries are to be found

agentName

public java.lang.String agentName

the name of the agent for which you want to analyse the annotations. If you want to analyse an interaction coding, this parameter should be null

Constructor Detail

SegmentBasedInspection

public SegmentBasedInspection(java.lang.String c,
                              java.lang.String o,
                              java.lang.String codingName,
                              java.lang.String segmentsLayer,
                              java.lang.String segmentElementName,
                              java.lang.String agentName,
                              NOMElementToTextDelegate segmentToText,
                              Predicate isIgnoreP,
                              NOMElementToValueDelegate segmentToValue,
                              DistanceMetric labelDistanceMetric,
                              double thMin,
                              double thMax,
                              int thSteps)

Method Detail

collectSegments

public void collectSegments()

Collect the segments for all annotators. Also initializes allAnnotatorPairs,allUsedAnnotators,actualNoOfAnn.

collectAlignments

public void collectAlignments(double thMin,
                              double thMax,
                              int thSteps)

Collect for all threshold values the alignments for all annotator pairs, using the SegmentAligner.

collectClassifications

public void collectClassifications()

Collect for all threshold values the derived classifications for all alignments, using the SegmentAlignmentToClassificationFactory.

collectMatrices

public void collectMatrices()

Collect for all Classification pairs the confusion and coincidence matrices used for calculation of reliability and inspection of confusions.

initReportPanel

public void initReportPanel()

renderSegments

public void renderSegments()

renderRelations

public void renderRelations()

drawLegend

public void drawLegend()

showConfusionTables

public void showConfusionTables()

Show the confusion tables in internal frames on the desktop

getCorpusName

public java.lang.String getCorpusName()

getObservationName

public java.lang.String getObservationName()

getCorpus

public NOMWriteCorpus getCorpus()

getMetaData

public NiteMetaData getMetaData()

getClock

public Clock getClock()

getClockFace

public ClockFace getClockFace()

Returns the clockface of this class. A clockface is the 'play control panel' for the video and audio, and in this tool for the virtual meeting animation as well. Before you initialize this, you must have at least a mainframe, a desktop and an initialized corpus.

generateThresholdValues

public void generateThresholdValues(double thMin,
                                    double thMax,
                                    int thSteps)

search

public java.util.List search(java.lang.String query)