Skip to main content

PLAY Project: The Protocol

9 Emotion

9.1 Workflow

  1. Choose whether to code child or mom emotion pass first
  2. Score each pass according to definitions in Emotion Codes.

9.2 Child emotion pass

Datavyu Emotion Codes for Child

There are 3 columns associated with child emotion codes: id for child emotion, child emotion column, and a comments column for the child emotion coding.

Child emotion id Column

Column name: emo_id_child
Code: (<lab_id>, <childemo_coder>, <childemo_date>, <childemo_mins>, <relchildemo_coder>, <relchildemo_date>)

Code the emo_id_child column according to the definitions for coding the id column.

Child emotion Column

Column name: childemo
Code: (<emotion_pn>)

childemo General Orientation

This code captures the times that the child is clearly displaying positive or negative emotion through facial expressions. Times when the child is in neutral emotion are not marked. Bouts of emotion as scored as events, where the grey spaces between emotion events mean the child is neutral or emotion is unclear. Coders also mark times as “missing” when the child’s head is not visible on camera and there is no possibility that emotion could be coded. Emotion may never be coded based solely on vocal affect. That is, when the child’s face is clearly not visible, do not code positive or negative emotion. Coders are watching/tagging the duration of each positive or negative emotion event by marking onset/offset times. To determine emotion, coders are watching the child’s face, not vocal affect. However, coders should keep audio on while coding in order to help differentiate talking from bouts of facial affect. To determine if emotion is not codeable/missing, coders are watching for when the head fully moves out of the camera view. After completing the coding pass for the entire video, coders should rewatch the first 5 minutes in order to recheck their codes and ensure that they did not miss any positive or negative emotion bouts. Coders must do this because each child expresses positive and negative emotions differently, which coders become more attuned to after extensive experience coding for that child. Therefore, coders may have inaccurately coded earlier parts of the video.

childemo Operational Definitions
Positive (p)

Code ‘p’ when the child is clearly displaying positive emotion (e.g., smiling). Code based off of the face and not off of the voice. Look for raising of the corners of the mouth, grinning and showing the teeth, along with closing of the eyes because of the raised cheeks. If there is any doubt that the child is showing positive emotion, then do not begin the code. Positive emotion cannot be coded based on the voice alone. So positive emotion could not be scored when the child’s face is not visible. If there is a mismatch between facial expression and vocal affect (eg. child’s face is neutral but sounds like they are laughing), always defer to the facial expression (eg. do not code this as positive emotion bout).

Negative (n)

Code ‘n’ when the child is clearly displaying negative emotion (e.g., frowning, wincing). Code based off of the face and not off of the voice (unless in cases where the face is not visible). Look for lowering of the corners of the mouth, stretching and tautness of the lips, along with closing of the eyes because of furrowed brow. If there is any doubt that the child is showing negative emotion, then do not begin the code. Negative emotion cannot be coded based on the voice alone, therefore. So negative emotion should never be scored when the child’s face is not visible. If there is a mismatch between facial expression and vocal affect (eg. child’s face is neutral but sounds like they are whining), always defer to the facial expression (eg. do not code this as negative emotion bout).

Setting the Onset & Offset of emotion
Onset & Offset for ‘p’ or ‘n’

Onset of an emotion event is the first frame the child is clearly displaying positive/negative emotion through the face. When coding ‘n’ from voice during missing time, set the onset when the negative voice starts and end the ‘n’ code when the voice ends. The coder is looking to identify when any lay person would absolutely agree the child is showing positive/negative emotion based on the face. There is no criterion for how long an emotion event should be. It is easy for the coder to mark the first frame when they see clear positive/negative emotion, even if it ends a few frames later. Events that are later deemed “too brief” could be removed via scripting. For cases when emotion code begins directly after missing (i.e. , the head has not been visible and the first frame you can see the face the infant is clearly displaying positive/negative emotion): Code the offset of missing and onset of the emotion code as the first frame when the face reappears. Use the “0” key to set the onset of the emotion event and simultaneously set the offset of ‘missing’. We want to preserve the 1ms difference between ‘missing’ and the emotion code so we can know that that ‘missing’ event was ended because of the onset of an emotion code. Offset of a positive/negative emotion is the first frame the child is clearly back to a neutral emotion through the face. The coder is looking to identify when any lay person would absolutely agree the child is no longer showing any positive/negative emotion based on the face. If the child’s face returns to neutral or is missing for less than 5 frames (150 ms) during one emotion code (e.g. positive, then neutral for 4 frames, then back to positive), continue the ‘p’ or ‘n’ code. The coder would have to expend unneeded effort to identify and tag those offsets and onsets time. For cases when emotion code is ended by missing (i.e., the emotion event may or may not have ended, but the coder can no longer see any part of the child’s head): Code the offset as the first frame the head completely is visible (see ‘missing’ code below). Use the “0” key to set the offset of emotion and simultaneously code the onset of ‘missing’. We want to preserve the 1ms difference between the emotion code and ‘missing’ so we can know that that ‘missing’ event caused the offset of that emotion event.

Missing (.)

Code ‘.’ for missing when child’s entire head is out of view (head could be occluded by furniture or mom, head not on camera view, or whole child missing from view). There is no minimum duration for coding missing bouts. It’s too labor intensive to code every time the face is in or out of view, but we do know that when none of the head is not visible (from the neck up) coding emotion is impossible. So to capture when emotion is not codeable in the most streamlined way, we only code missing when the whole head is not visible. If the coder could see any part of the face or head, then do not code missing.

9.2.0.0.1 Setting the Onset & Offset of emotion:

Onset is the first frame in which the coder clearly cannot see any part of the head. Offset is the first frame in which the coder can see the head again. If the child’s head was ‘missing’ and then reappeared for less than 5 frames, don’t stop the ‘.’ code to mark those few frames.

How to Code the Child Emotion Column

  • Make sure that the audio is on while coding child emotion.
  • Set “JUMP-BACK-BY” key to 1 s.
  • Enable cell highlighting (cmd + shift +F). Enable temporal mode (cmd T).
  • Play with #8-PLAY in real time (1x speed) until the child changes to clear positive or clear negative emotion or the head is completely not visible. Focus on the child’s face and do not be distracted by what the child is saying or doing. Audio gives context and helps to distinguish talking from emoting.
  • Pause with #5-STOP once you have identified a clear change in emotion or that the child’s head is no longer visible at all. Shuttle back with #4-SHUTTLEBACK at 1/8-1/4x speed to identify the onset. Use the mouth and eyes as the guide to onset. Press ENTER to set the onset as the frame where any lay person would say that child is happy or sad. The coder may even feel happy or sad watching the child’s face; use this as a guide for onset.
  • Then hit #8-PLAY then #4-SHUTTLEBACK once to watch at 1/2x and look for the offset of that emotion or when the head comes completely back into view. For missing, if it seems like there may be a long stretch of missing (e.g. child has completely wandered out of the room) then watch at 1x or 2x speed. Pause when you identify the offset.
  • Hit #1-JOGBACK and #3-JOGFORWARD to tag the frame the child’s face is clearly not positive or not negative (returned to neutral) or the head is visible again to code emotion from.
  • Then return to real time (1x speed) with #8-PLAY to watch for the next event.

Check the final coded pass

When finished coding, run the script to check for typos in the child emotion column. The script will flag any occurrence of typos that arise from an error in setting the onset or offset (not coded, cell with no duration, overlapping cells), consecutive ‘p’ cells or consecutive ‘n’ that are less than 5 frames apart (and should be merged), and any invalid codes (any characters besides ‘p’, ‘n’, or ‘.’). Fix errors and run the script again. When all the errors have been resolved the script will inform that no errors are found.

9.3 Mom emotion pass

9.3.1 Datavyu emotion Codes for Mom

There are 3 columns associated with mom emotion codes: id for mom emotion, mom emotion column, and a comments column for the mom emotion coding.

Mom emotion id Column

Column name: emo_id_mom
Code: (<lab_id>, <momemo_coder>, <momemo_date>, <momemo_mins>, <relmomemo_coder>, <relmomemo_date>)

Code the emo_id_mom column according to the definitions for coding the id column.

Mom emotion Column

Column name: momemo
Code: (<emotion_pn>)

General Orientation

This code captures the times that the mom is clearly displaying positive or negative emotion through facial expressions. Times when the mom is in neutral emotion are not marked. Bouts of emotion as scored as events, where the grey spaces between emotion events mean the mom is neutral or emotion is unclear. Coders also mark times as “missing” when the mom’s head is not visible on camera and there is no possibility that emotion could be coded. Emotion may never be coded based solely on vocal affect. That is, when the mom’s face is clearly not clearly visible, do not code positive or negative emotion. Coders are watching/tagging the duration of each positive or negative emotion event by marking onset/offset times. To determine emotion, coders are watching the mom’s face, not vocal affect. However, coders should keep audio on while coding in order to help differentiate talking from bouts of facial affect. To determine if emotion is not codeable/missing, coders are watching for when the head fully moves out of the camera view. After completing the coding pass for the entire video, coders should rewatch the first 5 minutes in order to recheck their codes and ensure that they did not miss any positive or negative emotion bouts. Coders must do this because each mom expresses positive and negative emotions differently, which coders become more attuned to after extensive experience coding for that mom. Therefore, coders may have inaccurately coded earlier parts of the video.

momemoOperational Definitions
Positive (p)

Code ‘p’ when the mom is clearly displaying positive emotion (e.g., smiling). Code based off of the face and not off of the voice. Look for raising of the corners of the mouth, grinning and showing the teeth, along with closing of the eyes because of the raised cheeks. If there is any doubt that the mom is showing positive emotion, then do not begin the code. Positive emotion cannot be coded based on the voice alone. So positive emotion could not be scored when the mom’s face is absolutely not visible. If there is a mismatch between facial expression and vocal affect (eg. mom’s face is neutral but sounds like they are laughing), always defer to the facial expression (eg. do not code this as positive emotion bout).

Negative (n)

Code ‘n’ when the mom is clearly displaying negative emotion (e.g., frowning, wincing). Code based off of the face and not off of the voice (unless in cases where the face is not visible). Look for lowering of the corners of the mouth, stretching and tautness of the lips, along with closing of the eyes because of furrowed brow. If there is any doubt that the mom is showing negative emotion, then do not begin the code. Negative emotion cannot be coded based on the voice alone, therefore. So negative emotion should never be scored when the mom’s face is not visible. If there is a mismatch between facial expression and vocal affect (eg. mom’s face is neutral but sounds like they are whining), always defer to the facial expression (eg. do not code this as negative emotion bout).

9.3.1.0.1 Setting the Onset & Offset of emotion:

Onset of an emotion event is the first frame the mom is clearly displaying positive/negative emotion through the face. When coding ‘n’ from voice during missing time, set the onset when the negative voice starts and end the ‘n’ code when the voice ends. The coder is looking to identify when any lay person would absolutely agree the mom is showing positive/negative emotion based on the face. There is no criterion for how long an emotion event should be. It is easy for the coder to mark the first frame when they see clear positive/negative emotion, even if it ends a few frames later. Events that are later deemed “too brief” could be removed via scripting. For cases when emotion code begins directly after missing (i.e. , the head has not been visible and the first frame you can see the face the infant is clearly displaying positive/negative emotion): Code the offset of missing and onset of the emotion code as the first frame when the face reappears. Use the “0” key to set the onset of the emotion event and simultaneously set the offset of ‘missing’. We want to preserve the 1ms difference between ‘missing’ and the emotion code so we can know that that ‘missing’ event was ended because of the onset of an emotion code. Offset of a positive/negative emotion is the first frame the mom is clearly back to a neutral emotion through the face. The coder is looking to identify when any lay person would absolutely agree the mom is no longer showing any positive/negative emotion based on the face. If the mom’s face returns to neutral or is missing for less than 5 frames during one emotion code (e.g. positive, then neutral for 4 frames, then back to positive), continue the ‘p’ or ‘n’ code. The coder would have to expend unneeded effort to identify and tag those offsets and onsets time, since reliability does not need to be frame accurate. For cases when emotion code is ended by missing (i.e., the emotion event may or may not have ended, but the coder can no longer see any part of the mom’s head): Code the offset as the first frame the head completely is visible (see ‘missing’ code below). Use “0” key to set the offset of emotion and simultaneously code the onset of ‘missing’. We want to preserve the 1ms difference between the emotion code and ‘missing’ so we can know that that ‘missing’ event caused the offset of that emotion event.

9.3.1.0.2 Missing (.)

Code ‘.’ for missing when the mom’s entire head is out of view (head could be occluded by furniture or child, head not on camera view, or whole mom missing from view). It’s too labor intensive to code every time the face is in or out of view, but we do know that when none of the head is not visible (from the neck up) coding emotion is impossible. So to capture when emotion is not codeable in the most streamlined way, we only code missing when the whole head is not visible. If the coder could see any part of the face or head, then do not code missing.

9.3.1.0.2.1 Setting the Onset & Offset of emotion:

Onset is the first frame in which the coder clearly cannot see any part of the head. Offset is the first frame in which the coder can see the head again. If the mom’s head was ‘missing’ and then reappeared for less than 5 frames, don’t stop the ‘.’ code to mark those few frames.

9.3.2 How to Code the Mom Emotion Column

  • Make sure that the audio is on while coding mom emotion.
  • Set “JUMP-BACK-BY” key to 1 s.
  • Enable cell highlighting (cmd + shift +F). Enable temporal mode (cmd T).
  • Play with #8-PLAY in real time (1x speed) until the mom changes to clear positive or clear negative emotion or the face is clearly not visible. Focus on the mom’s face and do not be distracted by what the mom is saying or doing. Audio gives context and helps to distinguish talking from emoting.
  • Pause with #5-STOP once you have identified a clear change in emotion or that the mom’s head is no longer visible at all. Shuttle back with #4-SHUTTLEBACK at 1/8-1/4x speed to identify the onset. Use the mouth and eyes as the guide to onset. Press ENTER to set the onset as the frame where any lay person would say that mom is happy or sad. The coder may even feel happy or sad watching the mom’s face; use this as a guide for onset.
  • Then hit #8-PLAY then #4-SHUTTLEBACK once to watch at 1/2x and look for the offset of that emotion or when the head comes completely back into view. For missing, if it seems like there may be a long stretch of missing (e.g. mom is on a different side of the room from the child) then watch at 1x or 2x speed. Pause when you identify the offset.
  • Hit #1-JOGBACK and #3-JOGFORWARD to tag the frame the mom’s face is clearly not positive or not negative (returned to neutral) or the head is visible again to code emotion from.
  • Then return to real time (1x speed) with #8-PLAY to watch for the next event.

9.3.3 Check the final coded pass

When finished coding, run the script to check for typos in the mom emotion column. The script will flag any occurrence of typos that arise from an error in setting the onset or offset (not coded, cell with no duration, overlapping cells), consecutive ‘p’ cells or consecutive ‘n’ that are less than 5 frames apart (and should be merged), and any invalid codes (any characters besides ‘p’, ‘n’, or ‘.’). Fix errors and run the script again. When all the errors have been resolved the script will inform that no errors are found.