net.sourceforge.nite.datainspection.timespan
Class BoundaryBasedInspection

java.lang.Object
  net.sourceforge.nite.datainspection.timespan.BoundaryBasedInspection

public class BoundaryBasedInspection
extends java.lang.Object

Inspection tool for timeline segmentations (gapped or non gapped) that investigates whether two annotators identified the same segment boundaries. Boundaries detected by two annotators are taken to be the same if they are at most a (configurable) threshold 'th' apart. Kappa and alpha are calculated by giving a pair of such aligned boundaries the label 'true' for both annotators, and giving unaligned boundaries the label 'true' for the annotator who detected the boundary and 'false' for the other.

Boundary based inspection

Introduction

Recall the summary in the nite.datainspection package documentation: Data inspection consists of

1. finding out whether separate annotators identified the same Items (segments, units for labeling),
2. finding out whether comparable Items have been assigned the same Values (labels, classes, categories) by the annotators and
3. finding out where disagreement lies, i.e. what Values are confused with each other in what situations; what type of Items are most often NOT identified by both annotators at the same time; etc.
4. (Investigating the nature of the errors that annotators made, and deciding how important these errors are, given the use for which the annotations were created.)

The Boundary Based inspections are involved with one possible approach to step one: finding out whether separate annotators identified the same items or events in the data. In this analysis, the focus is on detection whether two annotators identified a change point or boundary at the same moment, i.e. a start or end of a segment. This is a sensible approach to take if the detection of boundaries is explicitly defined as a separate step in the annotation instructions (e.g. "The task consists of two parts: first, defining "cuts" (segmentation points) in the video of a person at places where you see a distinct change in the mental state of this person, and second, to fill in a form that describes each segment that is thus created."). The two images below show how a segmentation of the timeline can be interpreted in terms of boundaries rather than segments.

an example 'timespan' annotation for two annotators

an example derived 'boundary' annotation for two annotators

The second image represents the lists of Boundaries extracted by the BoundaryExtractor for the two annotators. To interpret this in terms of 'whether the annotators identified the same boundaries', the relation between those derived boundary annotations must be analysed. The image below shows some relevant information. The red lines mark where two annotators found the same boundary; the blue dots mark boundaries found by only one annotator.

the alignment of the derived boundaries.

Boundary agreement: aligning boundaries

The alignment of the derived boundaries shown earlier can to a certain extent be calculated automatically. Precision of the alignment is determined by the threshold 'th': boundaries Ba and Bb annotated by annotators A and B can be seen as 'the same boundary' when they are at most 'th' time apart. The percentage of boundaries that have been aligned is a measure for how much two annotators agreed on the identification of boundaries. For the example alignment above:

        Aligned: 14
        Unaligned A: 0 (100% aligned)
        Unaligned B: 4 (78% aligned)

This information is reported by the class BoundaryBasedInspection.

Precision

Varying threshold 'th' gives information about the precision with which two annotators identified the same boundaries. Note however that 'th' should be low enough compared to the segment lengths in the annotations.

        ================
        ===Aligning f vs e
        ---
        - th 0.0
        50 aligned boundaries 
         For ann1: 15%.
         For ann2: 18%.
        ---
        - th 0.2
        209 aligned boundaries 
         For ann1: 63%.
         For ann2: 78%.
        ---
        - th 0.4
        228 aligned boundaries 
         For ann1: 69%.
         For ann2: 85%.
        ---
        - th 0.6
        241 aligned boundaries 
         For ann1: 73%.
         For ann2: 90%.
        ---
        - th 0.8
        243 aligned boundaries 
         For ann1: 74%.
         For ann2: 91%.
        ---
        - th 1.0
        249 aligned boundaries 
         For ann1: 76%.
         For ann2: 93%.

The list above gives an example output obtained from the BoundaryBasedInspection tool, for the FOA layer of the AMI corpus, with several variations of 'th'. For this specific annotation, 50% of the identified segments were shorter than 1 second. This means that the higher alignment percentages in the list are at least hard to interpret, and in the worst case not really meaningful. To gain more insight in these numbers, one should therefore have a look at the visualisation of the boundary alignments for different values of 'th', to see whether the alignments make sense or not. Also, one should relate the range of 'th' that is tried out to the distribution of segment lengths in the annotation.

The following three images, obtained for the same AMI FOA annotation, show how setting 'th' too low will cause the alignment percentages to be unfairly low, and setting 'th' too high will lead to unfairly high alignment percentages. These images can be produced using the tool BoundaryBasedInspection.

automatic alignment for a too low threshold.

more adequate automatic alignment.

automatic alignment for a too high threshold: the threshold is higher than the average segment length.

In the end, if you have been able to determine a sensible value for 'th', this analysis can lead to an answer to the question "what percentage of boundaries have been found by both annotators?" (In this particular case, the answer should be "about 65% for annotator 1 and about 85% for annotator 2"). Furthermore, the range of values for 'th' for which an acceptable alignment is found gives information about the precision with which annotators have annotated the data. In this case, this seems to be somewhere between 0.2 and 0.4 seconds.

(Side remark. Note that this has an implication for your use of the data. If the precision is on a few tenths of seconds, you should not focus on learning frame-perfect automatic boundary recognition. Furthermore, your evaluation of your machine learning results should keep this margin when assessing whether a detected boundary is 'good'.)

Trying to calculate alpha agreement for the alignment

Often, people are interested more in the chance corrected agreement than in the percentual agreement. One way to calculate such chance agreement for the alignment is to take every boundary detected by at least one annotator and take it as an Item. The Value then is True for the annotator who found the boundary, and False for the annotator who did not find the boundary, or True for both if they both found this boundary, as indicated by the alignment. However, this analysis cannot represent the fact that two annotators agree a lot on all the places where no boundary should be annotated, and therefore leads to ridiculously low alpha values, often even very negative. For the FOA annotations mentioned above, the resulting alpha values using this method will turn out to be somewhere between -0.2 and -0.8 for

Boundary agreement: timeline discretization

A possible solution to the problem that the 'alignment analysis' does not take into consideration the amount by which annotators agree that there is no boundary is to discretize the time line into short, equal-sized segments (length 'l'), and look for each annotator whether (s)he did note a boundary within that segment. The segment size 'l' is a measure of how much precision is required: a low 'l' means that boundaries are only seen as 'detected by both annotators' when the boundaries are very close together. This leads to a derived Classification where Items are each a short span on the timeline (TimespanItem) and Values are True or False (BooleanValue) depending on whether the given annotator has noted a boundary in that segment. These classifications can then be used to calculate CoincidenceMatrices and standard reliability measures such as kappa or alpha.

This is not a good idea!

The relative distributions of the labels true and false are extremely dependent on 'l': if 'l' gets smaller, the number of 'false' Values rises dramatically, whereas the number of 'true' Values is stable. This affects the outcome of the alpha and kappa values: they become higher when 'l' becomes smaller. On the other hand, smaller values for 'l' mean that less boundaries end up aligned (because a higher precision is required when 'l' is smaller), which means that kappa and alpha go down. And these two effects of course interact.

Conclusion: don't use that particular analysis (even though it was implemented in the class BoundaryBasedInspection2).

Boundary based inspection: Conclusion

All in all, the class BoundaryBasedInspection and its supporting classes provide support for determining a sensible threshold for boundary alignment as well as for performing this alignment. In the end, the percentual agreement on the occurrence of boundaries gives an indication of how often annotators find the same boundaries, and the determined threshold value gives an indication of how precise the timing of the annotations is.

Field Summary

java.lang.String	agentName the name of the agent for which you want to analyse the annotations.
java.lang.String	codingName the name of the Coding in which the boundaries are to be found
java.lang.String	commonLayer the name of the common layer shared by all annotators, can be null
java.lang.String	corpusName corpus
java.lang.String	observationName observation
java.lang.String	segmentElementName the name of the Elements in the Layer in that Coding in which the boundaries are to be found
java.lang.String	segmentsLayer the name of the Layer in that Coding in which the boundaries are to be found

Constructor Summary

BoundaryBasedInspection(java.lang.String c, java.lang.String o, java.lang.String codingName, java.lang.String segmentsLayer, java.lang.String segmentElementName, java.lang.String agentName, double thMin, double thMax, int thSteps)

BoundaryBasedInspection(java.lang.String c, java.lang.String o, java.lang.String codingName, java.lang.String segmentsLayer, java.lang.String commonLayer, java.lang.String segmentElementName, java.lang.String agentName, double thMin, double thMax, int thSteps)

Method Summary

void	collectAlignments(double thMin, double thMax, int thSteps) Collect for all threshold values the alignments for all annotator pairs, using the BoundaryAligner.
void	collectBoundaries() Collect the derived Boundary annotations for all annotators.
void	collectClassifications() Collect for all threshold values the derived classifications for all alignments, using the BoundaryAlignmentToClassificationFactory.
void	collectMatrices() Collect for all Classification pairs the confusion and coincidence matrices used for calculation of reliability and inspection of confusions.
void	generateThresholdValues(double thMin, double thMax, int thSteps)
Clock	getClock()
ClockFace	getClockFace() Returns the clockface of this class.
NOMWriteCorpus	getCorpus()
java.lang.String	getCorpusName()
NiteMetaData	getMetaData()
java.lang.String	getObservationName()
void	initReportPanel()
void	renderBoundaries()
void	renderRelations()
java.util.List	search(java.lang.String query)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

corpusName

public java.lang.String corpusName

corpus

observationName

public java.lang.String observationName

observation

codingName

public java.lang.String codingName

the name of the Coding in which the boundaries are to be found

segmentsLayer

public java.lang.String segmentsLayer

the name of the Layer in that Coding in which the boundaries are to be found

commonLayer

public java.lang.String commonLayer

the name of the common layer shared by all annotators, can be null

segmentElementName

public java.lang.String segmentElementName

the name of the Elements in the Layer in that Coding in which the boundaries are to be found

agentName

public java.lang.String agentName

the name of the agent for which you want to analyse the annotations. If you want to analyse an interaction coding, this parameter should be null

Constructor Detail

BoundaryBasedInspection

public BoundaryBasedInspection(java.lang.String c,
                               java.lang.String o,
                               java.lang.String codingName,
                               java.lang.String segmentsLayer,
                               java.lang.String commonLayer,
                               java.lang.String segmentElementName,
                               java.lang.String agentName,
                               double thMin,
                               double thMax,
                               int thSteps)

BoundaryBasedInspection

public BoundaryBasedInspection(java.lang.String c,
                               java.lang.String o,
                               java.lang.String codingName,
                               java.lang.String segmentsLayer,
                               java.lang.String segmentElementName,
                               java.lang.String agentName,
                               double thMin,
                               double thMax,
                               int thSteps)

Method Detail

collectBoundaries

public void collectBoundaries()

Collect the derived Boundary annotations for all annotators. Also initializes allAnnotatorPairs,allUsedAnnotators,actualNoOfAnn.

collectAlignments

public void collectAlignments(double thMin,
                              double thMax,
                              int thSteps)

Collect for all threshold values the alignments for all annotator pairs, using the BoundaryAligner.

collectClassifications

public void collectClassifications()

Collect for all threshold values the derived classifications for all alignments, using the BoundaryAlignmentToClassificationFactory.

collectMatrices

public void collectMatrices()

Collect for all Classification pairs the confusion and coincidence matrices used for calculation of reliability and inspection of confusions.

initReportPanel

public void initReportPanel()

renderBoundaries

public void renderBoundaries()

renderRelations

public void renderRelations()

getCorpusName

public java.lang.String getCorpusName()

getObservationName

public java.lang.String getObservationName()

getCorpus

public NOMWriteCorpus getCorpus()

getMetaData

public NiteMetaData getMetaData()

getClock

public Clock getClock()

getClockFace

public ClockFace getClockFace()

Returns the clockface of this class. A clockface is the 'play control panel' for the video and audio, and in this tool for the virtual meeting animation as well. Before you initialize this, you must have at least a mainframe, a desktop and an initialized corpus.

generateThresholdValues

public void generateThresholdValues(double thMin,
                                    double thMax,
                                    int thSteps)

search

public java.util.List search(java.lang.String query)