Tracking Rats in Operant Conditioning Chambers Using a Versatile Homemade Video Camera and DeepLabCut

Video camera performance

The representative results were gathered in operant conditioning chambers for rats with floor areas of 28.5 cm x 25.5 cm, and heights of 28.5 cm. With the fisheye lens attached, the camera captures the full floor area and large parts of the surrounding walls, when placed above the chamber (Figure 7A). As such, a good view can be obtained, even if the camera is placed off-center on the chamber’s top. This should hold true for comparable operant chambers. The IR LEDs are able to illuminate the entire chamber (Figure 7B,C), enabling a good view, even when all other lights inside the chamber are switched off (Figure 7C). However, the lighting in such situations is not entirely even, and may result in some difficulties in obtaining accurate tracking. If such analysis is of interest, additional sources of IR illumination might be required. It is also worth noting that some chambers use metal dropping pans to collect urine and feces. If the camera is placed directly above such surfaces, strong reflections of the IR LEDs’ light will be visible in the recorded videos (Figure 7B). This can, however, be avoided by placing paper towels in the dropping pan, giving a much-improved image (Figure 7C). Placing the camera’s IR or colored LEDs too close to the camera lens may result in them being visible in the image periphery (Figure 7B). As the camera is IR sensitive, any IR light sources that are present inside the chambers may be visible in the videos. For many setups, this will include the continuous shining of IR beam break sensors (Figure 7C). The continuous illumination from the camera’s IR LEDs does not disturb the image quality of well-lit chambers (Figure 7D). The size of the videos recorded with the camera is approximately 77 Mb/min. If a 32 Gb micro SD card is used for the camera, there should be about 20 Gb available following the installation of the operating system. This leaves room for approximately 260 min of recorded footage.

The fisheye lens causes the camera to have a slightly uneven focus, being sharp in the center of the image but reduced sharpness towards the edges. This does not appear to affect the accuracy of tracking. Moreover, the fisheye lens results in the recorded image being distorted. For example, the distances between equally spaced points along straight lines will show artificially reduced spacing towards the periphery of the image (Figure 9A,B). If the camera is used for applications where most of the field of view or absolute measurements of distance and speed are of interest, it is worth considering correcting the data for this distortion²³ (Supplementary File 4). The distortion is, however, relatively mild in the center of the image (Figure 9B). For videos gathered in our operant chamber, the area of interest is limited to the central 25% of the camera’s field of view. Within this area, the effect of the fisheye distortion is minimal (Figure 9C‒F).

Accuracy of tracking with DeepLabCut

The main factors that will determine the tracking accuracy of a trained network are (i) the number of labeled frames in its training data set, (ii) how well those labeled frames capture the behavior of interest and (iii) the number of training iterations used. DeepLabCut includes an evaluate function, which reports an estimate of how far away (in numbers of pixels) its tracking can be expected to be from the actual location of an object. This, however, does not necessarily give a good description of the number of frames where an object is lost and/or mislabeled (Figure 10A), prompting the need for additional manual assessment of tracking accuracy.

For analyzing behaviors inside an operant chamber, a well-trained network should allow the accurate identification of all events where the protocol step indicator is active. If not, retraining the network or choosing a different indicator might be needed. Despite having a well-trained network, tracking of the protocol step indicator may on occasion be disrupted by animals blocking the camera’s view (Figure 10B). This will cause breaks in the tracking that are reminiscent of episodes where the indicator is inactive. The frequency of this happening will depend on animal strain, type of behavioral protocol and choice of protocol step indicator. In the example data from the 5CSRTT that is used here, it occurred on four out of 400 trials (data not shown). All occasions were easily identifiable, as their durations did not match that of the break step that had been included in the protocol design (Figure 6A). Ultimately, choosing an indicator that is placed high up in the chamber and away from components that animals interact with is likely to be helpful.

A well-trained network should allow >90% accuracy when tracking an animal’s head during video segments of interest (Video 1). With this, only a small subset of video frames will need to be excluded from the subsequent analysis and usable tracking data will be obtainable from virtually all trials within a test session. Accurate tracking is clearly identifiable by markers following an animal throughout its movements (Video 2) and plotted paths appearing smooth (Figure 10C). In contrast, inaccurate tracking is characterized by markers not reliably staying on target (Video 3) and by plotted paths appearing jagged (Figure 10D). The latter is caused by the object being tracked to distant erroneous positions in single video frames within sequences of accurate tracking. As a result of this, inaccurate tracking typically causes sudden shifts in calculated movement speeds (Figure 10E). This can be used to identify video frames where tracking is inaccurate, to exclude them from subsequent analysis. If there are substantial problems with tracking accuracy, the occasions where tracking fails should be identified and the network should be retrained using an expanded training set containing labeled video frames of these events (Figure 10A,E).

Use of video tracking to complement analysis of operant behaviors

Analyzing how an animal moves and positions itself during operant tests will provide multiple insights into the complex and multifaceted nature of their behaviors. By tracking where an animal is located throughout a test session, one can assess how distinct movement patterns relate to performance (Figure 11A,B). By further investigating head movements during specific protocol steps, one can detect and characterize the use of different strategies (Figure 11C‒E).

To exemplify, consider the representative data presented for rats performing the 5CSRTT test (Figure 6A, Figure 11). In this test, animals are presented with multiple trials that each start with a 5 s waiting step (inter-trial interval – ITI) (Figure 6A: 1). At the end of this, a light will shine inside one of the nose poke openings (randomly chosen position on each trial, Figure 6A: 2). Nose-poking into the cued opening is considered a correct response and is rewarded (Figure 6A: 3). Responding into another opening is considered incorrect. Failing to respond within 5 s following the presentation of the light is considered an omission. Tracking head movements during the ITI of this test has revealed that on trials where rats perform a response, they are fast at moving towards the area around the nose poke openings (Figure 11A,B, Video 4). In contrast, on the majority of omission trials, the rats fail to approach the area around the openings (Figure 11B, Video 5). This behavior is in line with the common interpretation of omissions being closely related to a low motivation to perform the test^3,16. However, on a subset of omission trials (approximately 20% of the current data set), the rats showed a clear focus towards the openings (Figure 11B, Video 6) but failed to note the exact location of the cued opening. The data thus indicate that there are at least two different types of omissions, one related to a possible disinterest in the ongoing trial, and another that is more dependent on insufficient visuospatial attention³. Head tracking can also be used to distinguish apparent strategies. As an example, two distinct attentional strategies were revealed when analyzing how the rats move when they are in proximity to the nose poke openings during the 5CSRTT (Figure 11C‒E). In the first strategy, rats showed an extremely focused approach, maintaining a central position throughout most of the ITI (Figure 11C, Video 7). In contrast, rats adopting the other strategy constantly moved their heads between the different openings in a search-like manner (Figure 11D, Video 8). This type of behavioral differences can conveniently be quantified by calculating the amount of time spent in proximity to the different openings (Figure 11E). Finally, by analyzing which opening the rat is closest to at the time of cue light presentation (Figure 11F), it can be demonstrated that being in a central position (Figure 11G) and/or in close proximity to the location of the cued opening (Figure 11H) seems to be beneficial for accurate performance on the test.

Figure 1: Sketch of the listed microcomputer. The schematic shows the position of several components of interest on the microcomputer motherboard. These are marked with circled numbers as follows: 1: Connector for camera ribbon cable; 2: LED light indicating when computer is running; 3: Micro USB for power cable; 4: Micro USB for mouse/keyboard; 5: General-purpose input/output pins (GPIO pins), these pins are used to connect the microcomputer to LEDs, switches, and the IR LED module; 6: Mini HDMI output; 7: Micro SD card slot. In the lower portion of the figure, a cropped and enlarged part of the GPIO pins is shown to indicate how to count along them to correctly identify the position of a specific pin. Please click here to view a larger version of this figure.

Figure 2: Building the main body of the camera. The figures illustrated the main steps for building the body of the camera. (A) Attach the magnetic metal ring to the camera stand. (B) Attach the camera module to the camera stand. (C) Connect the camera module to the microcomputer via the flat ribbon cable. Note the white arrows indicating how to open and close the camera ports present on both the microcomputer and the camera module. (D) Place the microcomputer into the plastic casing and insert a micro SD card. Please click here to view a larger version of this figure.

Figure 3: Updating the microcomputer’s operating system and enabling the peripherals. The figure shows four different screenshots depicting the user interface of the microcomputer. (A) Terminal windows can be opened by clicking the “terminal” icon in the top left corner of the screen. (B) Within the terminal, one can type in different kinds of commands, as detailed in the protocol text. The screenshot displays the command for updating the system’s software packages. (C) The screenshot displays how to navigate to the configurations menu, where one can enable the use of the camera module and the I2C GPIO pins. (D) The screenshot displays the /home/pi folder, where the camera script should be copied in step 1.10 of the protocol. The window is opened by clicking the indicated icon in the top left corner of the screen. Please click here to view a larger version of this figure.

Figure 4: Configuring the microcomputer’s rc.local file. The figure displays two screenshots of the microcomputer’s rc.local file, when accessed through the terminal as described in step 1.11.1. (A) A screenshot of the rc.local file in its original format. The arrow indicates the space where text needs to be entered in order to enable the auto-start feature of the camera. (B) A screenshot of the rc.local file after it has been edited to shine the IR LEDs and start the python script controlling the camera upon startup of the microcomputer. Please click here to view a larger version of this figure.

Figure 5: Connecting of switches and LEDs to microcomputer’s GPIO pins. (A) Schematic showing a button switch with female jumper cables (top) and a LED with resistor and female jumper cables (bottom). (1) Button switch, (2) female jumper cables, (3) LED, (4) resistor. (B) Schematic image showing how the two button switches, the colored LEDs and the IR LED board are connected to the GPIO pins of the microcomputer. Blue cables and GPIO pins indicate ground. The position of two GPIO pins are indicated in the figure (GPIO pins #2 and #40): (1) Button for starting/stopping video recording. (2) LED indicating when video is being recorded. (3) Button for switching off camera. (4) LED indicating when the camera has booted and is ready to be used. (5) IR LED module. Note that circuits with LEDs also contain 330 Ω resistors. Please click here to view a larger version of this figure.

Figure 6: Using DeepLabCut tracking of protocol step indicator to identify sequences of interest in full-length videos. (A) Schematic of the steps of a single trial in the 5-choice serial reaction time test (5CSRTT): (1) First, there is a brief waiting period (ITI). Arrow indicates an actively shining house light. (2) At the end of the ITI, a light will shine in one of the five nose poke openings (arrow). (3) If a rat accurately responds by performing a nose poke into the cued opening, a reward is delivered (arrow). (4) The rat is allowed to retrieve the reward. (5) To enable the use of the house light as a protocol step indicator, a brief pause step where the house light is switched off (arrow) is implemented before the next trial begins. Note that the house light is shining during subsequent steps of the trial. (B) An example graph depicting the x-coordinate of the active house light, as tracked by DeepLabCut, during a video segment of a 5CSRTT test. During segments where the house light is shining (indicator active – 1), the position is tracked to a consistent and stable point (also note the red marker (indicated by arrow) in the example video frame), comparable to that of the house light’s position in Figure 8C (x, y: 163, 503). During segments where the house light is not shining (indicator inactive – 2, note the white arrow in the example video frame), the tracked position is not stable, and far away from the house light’s actual coordinate. (C) Table 1 shows an example of processed output obtained from DeepLabCut tracking of a protocol step indicator. In this output, the starting point for each occasion where the indicator is active has been listed. Table 2 depicts an example of data obtained from the operant conditioning system, giving relevant details for individual trials. In this example, the duration of the ITI, position of the cued opening and latencies to perform a response and retrieve the reward have been recorded. Table 3 depicts an example of data obtained by merging tracking results from DeepLabCut and data recorded from the operant conditioning system. Through this, the video frames for the starting point of the ITI (step 1 in A), the starting point of the cue light presentation (step 2 in A), the response (step 3 in A) and retrieval (step 4 in A) for an example trial have been obtained. (D) An example graph depicting the x-coordinate of the house light, as tracked by DeepLabCut, during a filmed 5CSRTT trial. The different steps of the protocol are indicated: (1) ITI; (2) presentation of a cue light (position indicated by white arrow); (3) response; (4) reward retrieval. The identification of video frames depicting the start and stop of these different protocol steps was done through a process comparable to that indicated in D. Please click here to view a larger version of this figure.

Figure 7: Image characteristics of camera. (A) Uncropped image obtained from a camera placed on top of an operant conditioning chamber. The image was captured while the chamber was placed in a brightly lit room. Note the (1) house light and (2) reward pellet trough along the chamber’s left wall and (3) the row of five nose poke openings along the chamber’s right wall. Each nose poke opening contains a small cue light. (B) Uncropped image displaying the strong reflection caused by (1) the metal dropping pan, as well as reflections caused by sub-optimal positioning of the camera’s (2) indicator LEDs and (3) IR LED module. (C) Cropped image of the chamber in complete darkness. Note that the lights from the IR beam break detectors in the five nose poke openings along the chamber’s right wall are clearly visible (arrow). (D) Cropped image of the chamber when brightly lit. Please click here to view a larger version of this figure.

Figure 8: Positional tracking of protocol step indicator and body parts of interest. (A) The picture shows the position of a protocol step indicator (red) as well as the head (yellow) and tail (green) of a rat, as tracked by DeepLabCut. As indicated by the tracking of the lit house light, the video frame is taken from a snapshot of an active trial. (B) The picture shows the position of the head (yellow) and tail (green) as tracked by DeepLabCut during a moment when a trial is not active. Note the lack of house light tracking. (C) The positions of points of interest used in the analysis of data shown in Figure 6 and Figure 11; (1) House light, in this case used as protocol step indicator, (2‒6) Nose poke openings #1‒5. Please click here to view a larger version of this figure.

Figure 9: Image distortion from fisheye lens. (A) Image of a checker-board pattern with equally sized and spaced black and white squares taken with the camera described in this protocol. Image was taken at a height comparable to that used when recording videos from operant conditioning chambers. Black squares along the central horizontal and vertical lines have been marked with DeepLabCut. (B) Graph depicting how the spacing of the marked squares in (A) change with proximity to the image center. (C) Image depicting measurements taken to evaluate impact of fisheye distortion effect on videos gathered from operant chambers. The corners and midpoints along the edge of the floor area, the central position of each individual floor rung and the position of the five nose poke openings have been indicated with DeepLabCut (colored dots); (1) spacing of floor rungs, (2) width of chamber floor along the middle of the chamber, (3) spacing of nose poke openings. (D) Spacing of floor rungs (averaged for each set of three consecutive rungs), numbered from left to right in (C). There is a small effect of the fisheye distortion, resulting in the central rungs being spaced roughly 3 pixels (8%) further apart than rungs that are positioned at the edges of the chamber floor. (E) Width of the chamber floor in (C) measured at its left and right edges, as well as midpoint. There is a small effect of the fisheye distortion, resulting in the width measured at the midpoint being roughly 29 pixels (5%) longer than the other measurements. (F) Spacing of nose poke openings in (C), numbered from the top of the image. There is a small effect of the fisheye distortion, resulting in the spacing between the central three openings (H2, H3, H5) being roughly 5 pixels (4%) broader than the spacing between H1-H2 and H4-H5. For D-F, data were gathered from four videos and graphs depict group mean + standard error. Please click here to view a larger version of this figure.

Figure 10: Reviewing accuracy of DeepLabCut tracking. (A) A table listing training information for two neural networks trained to track rats within operant chambers. Network #1 used a smaller training data set, but high number of training iterations compared to Network #2. Both networks achieved a low error score from DeepLabCut’s evaluation function (DLC test error) and displayed a low training loss towards the end of the training. Despite this, Network #1 showed very poor tracking accuracy upon manual evaluation of marked video frames (measured accuracy, estimated from 150 video frames covering a video segment comparable to those in Video 2 and Video 3). Network #2 represents the improved version of Network #1, after having included additional video frames of actively moving rats into the training data set, as described in (E). (B) Image depicting a rat rearing up and covering the chamber’s house light (Figure 7A) with its head, disrupting the tracking of it. (C) Video frame capturing a response made during a 5CSRTT trial (Figure 6A: 3). The head’s movement path during the response and preceding ITI has been superimposed on the image in yellow. The tracking is considered to be accurate. Note the smooth tracking during movements (white arrow). A corresponding video is available as Video 2. Network #2 (see A) was used for tracking. (D) Video frame capturing a response made during a 5CSRTT trial (Figure 6A: 3). The head’s movement path during the response and preceding ITI has been superimposed on the image in yellow. Data concerns the same trial as shown in (C) but analyzed with Network #1 (see A). The tracking is considered to be inaccurate. Note the path’s jagged appearance with multiple straight lines (white arrows), caused by occasional tracking of the head to distant erroneous positions (black arrows). A corresponding video is available as Video 3. (E) Graph depicting the dynamic changes in movement speed of the head tracking in (C) and (D). Identifiable in the graph are three major movements seen in Video 2 and 3, where the rat first turns to face the nose poke openings (initial turn), makes a small adjustment to further approach them (adjustment), and finally performs a response. The speed profile for the good tracking obtained by Network #2 (A) displays smooth curves of changes in movement speed (blue arrows), indicating an accurate tracking. The speed profile for the poor tracking obtained by Network #1 (A) shows multiple sudden spikes in movement speed (red arrows) indicative of occasional tracking errors in single video frames. It is worth noting that these tracking problems specifically occur during movements. To rectify this, the initial training set used to train Network #1 was expanded with a large amount of video frames depicting actively moving rats. This was subsequently used to train Network #2, which efficiently removed this tracking issue. Please click here to view a larger version of this figure.

Figure 11: Use of positional tracking through DeepLabCut to complement the behavioral analysis of operant conditioning tests. (A) A top view of the inside of an operant conditioning chamber. Three areas of the chamber are indicated. The area close to the reward pellet trough (Pellet), the central chamber area (Center) and the area around the nose poke openings (Openings). (B) A graph depicting the relative amount of time rats spend in the three different areas of the operant chamber outlined in (A) during the ITI step of the 5CSRTT. Note that on trials with a response, rats initially tend to be positioned close to the pellet trough (black) and chamber center (grey), but as the ITI progresses, they shift towards positioning themselves around the nose poke openings (white). In contrast, on typical omission trials, rats remain positioned around the pellet trough and chamber center. On a subset of omission trials (approximately 20%) rats clearly shift their focus towards the nose poke openings, but still fail to perform a response when prompted. Two-way ANOVA analysis of the time spend around the nose poke openings using trial type as between-subject factor and time as within-subject factor reveal significant time (p < 0.001, F(4,8) = 35.13), trial type (p < 0.01, F(2,4) = 57.33) and time x trial type interaction (p < 0.001, F(8,16) = 15.3) effects. Data gathered from four animals performing 100 trials each. Graph displays mean + standard error. (C) Heat map displaying all head positions tracked in proximity of the nose poke openings, by one specific rat during 50 ITIs of a 5CSRTT test session. Note that the rat has a strong tendency to keep its head in one spot close to the central nose poke opening. (D) Heat map displaying all head positions tracked in proximity of the nose poke openings, by one specific rat during 50 ITIs of a 5-CSRTT test session. Note that the rat shows no clear preference for any specific opening. (E) Graph depicting the relative amount of time that the two rats displayed in (C) and (D) spend being closest to the different nose poke openings during 50 ITIs of the 5CSRTT. The rat displaying a focused strategy (C) (black) shows a strong preference for being closest to the central opening while the rat with a search-like strategy (D) (white) shows no preference for any particular opening. The graph depicts average + standard error. (F) An image of a rat at the time of cue presentation on a 5CSRTT trial (Figure 6A). Note that the rat has positioned its head closest to the central opening (white arrow), being two openings away from the cued opening (black arrow). (G) A graph depicting performance accuracy on the 5CSRTT (i.e., frequency of performing correct responses) in relation to whether the head of the rats was closest to the central opening or one of the other openings at the time of cue presentation (Figure 6A2). Data gathered from four animals performing roughly 70 responses each. Graph displays group mean + standard error (matched t-test: p < 0.05). (H) A graph depicting performance accuracy on the 5CSRTT in relation to the distance between the position of the cued opening and the position of a rat’s head, at the point of signal presentation. The distance relates to the number of openings between the rats’ head position and the position of the signaled opening. Data gathered from four animals performing roughly 70 responses each. Graph displays mean + standard error (matched one-way ANOVA: p < 0.01). For the presented analysis, Network #2 described in Figure 10A was used. The complete analyzed data set included approximately 160,000 video frames and 400 trials. Out of these, 2.5% of the video frames were excluded due the animal’s noted movement speed being above 3,000 pixels/s, indicating erroneous tracking (Figure 10E). No complete trials were excluded. Please click here to view a larger version of this figure.

Video 1: Representative tracking performance of well-trained neural network. The video shows a montage of a rat performing 45 trials with correct responses during a 5CSRTT test session (see Figure 6A for protocol details). Tracking of the house light (red marker), tail base (green marker) and head (blue marker) are indicated in the video. The training of the network (Network #2 in Figure 10A) emphasized accuracy for movements made along the chamber floor in proximity to the nose poke openings (right wall, Figure 7A). Tracking of these segments show on average >90% accuracy. Tracking of episodes of rearing and grooming are inaccurate, as the training set did not include frames of these behaviors. Note that the video has been compressed to reduce file size and is not representable of the video quality obtained with the camera. Please click here to download this video.

Video 2: Example of accurately tracked animal. The video shows a single well-tracked trial of a rat performing a correct response during the 5CSRTT. Tracking of the house light (red marker), tail base (green marker) and head (blue marker) are indicated in the video. Neural network #2 described in Figure 10A was used for tracking. Note how the markers follow the movements of the animal accurately. Also refer to Figure 10C,E for the plotted path and movement speed for the head tracking in this video clip. Please click here to download this video.

Video 3: Example of poorly tracked animal. The video shows a single poorly tracked trial of a rat performing a correct response during the 5CSRTT. Tracking of the house light (red marker), tail base (green marker) and head (blue marker) are indicated in the video. Neural network #1 described in Figure 10A was used for tracking. The video clip is the same as the one used in Video 2. Note that the marker for the head is not reliably placed on top of the rat’s head. Also refer to Figure 10D,E for the plotted path and movement speed for the head tracking in this video clip. Please click here to download this video.

Video 4: Example of movements made during a 5CSRTT trial with a response. The video shows a single well-tracked trial of a rat performing a correct response during the 5-CSRTT. Tracking of the house light (red marker), tail base (green marker) and head (blue marker) are indicated in the video. Note how the rat at first is positioned in clear proximity to the pellet receptacle (left wall, Figure 7A) and then moves over to focus its attention on the row of nose poke openings. Please click here to download this video.

Video 5: Example of a typical omission trial during the 5CSRTT. The video shows a single well-tracked trial of a rat performing a typical omission during the 5CSRTT. Tracking of the house light (red marker), tail base (green marker) and head (blue marker) are indicated in the video. Note how the rat maintains its position around the pellet receptacle (left wall, Figure 7A) and chamber center, rather than turning around to face the nose poke openings (right wall, Figure 7A). The displayed behavior and cause of the omission can be argued to reflect low interest in performing the test. Please click here to download this video.

Video 6: Example of an atypical omission trial during the 5CSRTT. The video shows a single well-tracked trial of a rat performing an atypical omission during the 5CSRTT. Tracking of the house light (red marker), tail base (green marker) and head (blue marker) are indicated in the video. Note how the rat positions itself towards the nose poke openings along the right wall of the chamber (Figure 7A). This can be argued to indicate that the animal is interested in performing the test. However, the rat faces away from the cued opening (central position) when the cue is presented (5 s into the clip). In contrast to the omission displayed in Video 4, the one seen here is likely related to sub-optimal visuospatial attention processes. Please click here to download this video.

Video 7: Example of an animal maintaining a focused central position during an ITI of the 5CSRTT. The video shows a single well-tracked trial of a rat performing a correct response on a trial of the 5CSRTT. Note how the rat maintains a central position during the ITI, keeping its head steady in proximity to the central nose poke opening along the chambers right wall (Figure 7A). Tracking of the house light (red marker), tail base (green marker) and head (blue marker) are indicated in the video. Please click here to download this video.

Video 8: Example of an animal displaying a search-like attentional strategy during an ITI of the 5CSRTT. The video shows a single well-tracked trial of a rat performing a correct response on a trial of the 5CSRTT. Note how the rat frequently repositions its head to face different nose poke openings along the right wall of the chamber (Figure 7A). Tracking of the house light (red marker), tail base (green marker) and head (blue marker) are indicated in the video. Please click here to download this video.