Using an Automated 3D-tracking System to Record Individual and Shoals of Adult Zebrafish

All experiments were conducted in accordance with the National Institute of Health Guide for Care and Use of Laboratory Animals.

1. Calibration

1. Place an empty observation container in the chamber so that the feet of the container are inserted into the holes provided for correct alignment. Orient the container such that the semi-transparent walls (the purpose of their structure is to reduce reflections) face the far side and to the right of the camera.
2. Open the Record Module, select 'Camera control' and adjust 'gain' and 'shutter' (after deselecting them, boxes on the right) so that the image is clear. Place the side mirror in an approximately 45° angle at the left side of the observation container so that the side view of the container on the computer monitor is fully visible. Furthermore, the mirror has to be aligned such that the front wall of the container is seen as one line (as indicated by the arrow in Figure 3).
3. Connect the calibration panel to the parallel port (of the camera card) and press 'LED panel on/off' button in Record Module to switch the LEDs on.
4. Place the calibration panel in the empty observation container such that the 5 LEDs can be seen in the front, top and side views on the computer monitor as in Figure 3 (sometimes the cable blocks the view, in this case move it). Make also sure that the LEDs are below the marks for the water level (these marks are engraved in the observation containers at 135 mm from the inside bottom of the tank) and that the position of the panel is stable (so that it does not move when filling the container with water in step 1.9).
5. Switch off the lights in the observation chamber and shut the curtain (be careful not to touch the cable of the calibration panel or the side mirror).
6. Push the 'Data location' button and make or select a folder for the calibration files. These files will be used for all subsequent recordings (or experiments) until recalibration becomes necessary. Thus, ideally make the calibration data part of the name of the calibration folder.
7. Adjust the three red boxes on the computer screen such that each just encompasses the 5 LEDs in any of the three views (Figure 4, left panel). Again use 'gain' and 'shutter' under 'Camera control' to obtain clear images of the LEDs, without images of other light points (which are due to noise, reflections or water droplets on the wall of the container), as in Figure 4, right panel.
8. Switch the LEDs off and push the 'Collect calibration w/o water' button. On the computer screen 5 tags ('x') numbered from 0-4 appear. Switch the panel on. The locations of the tags should coincide with the LEDs. If not, repeat step 1.7 (see Figure 4 for illustration of incorrect tag assignment on left panel and correct tag assignment on right panel).
9. Fill the observation chamber with water up to the mark. The LEDs should be beneath water level (as was explained in step 1.4). If they are above water level, empty the container and go back to step 1.4. When filling the water into the container, be careful that the LED panel does not move. However, should this happen, then empty the container and go back to the step 1.4. This is important, because calibration is based on determining the differences between the recorded LEDs positions without and with water.
10. Push the 'Collect calibration w. water' button. Again, like in step 1.8, five tags will appear on the screen, which in most cases will be slightly offset from those obtained during calibration without water. In this step, gain and shutter might have to be readjusted. Repeat until the tags are placed correctly on top of the LEDs (as in Figure 4, right panel).
11. Switch lights on in the observation chamber, adjust gain and shutter (under 'Camera Control') so that the image is not overly bright.
12. Figure 5 shows the six points that have to be selected sequentially by right-click the mouse. Be sure that the points are on the designated lines: 1. front view water level, 2. left top ridge of container in top view, 3. vertical line that outlines the front wall in side view. The length of the line is not essential. Between clicking positions 1 and 2, 3 and 4, and 5 and 6, straight lines will appear (the numbers will actual not appear on the screen). Press 'Collect container information' to save this information.
13. Remove both side mirror and LED panel and fill the water exactly up to the mark. Select 'Start Recording'. A new window asks for the duration in minutes. For the current purpose 1 min is enough. Press OK.
14. After recording, exit Record Module, open Trajectory Module, select the calibration folder (made in step 1.6) with 'Data Location' (loading of the file might take some time). This folder contains the calibration files and the files of the just recorded data (step 1.13).
15. Press the 'Acquire Background' button. A new window opens, without changing anything press OK. Wait until a (upside-down) image of the observation tank appears on the monitor. Then press 'Load Background'. The background picture will be corrected. A new window opens that asks for adjusting the boxes. Simply press OK.
16. Push sequentially third through seventh button: 'Load Correspondence w/o water', 'Calibrate virtual camera(s)', 'Load refracted correspondence', 'Collect container geometries', 'Retrieve animal container info'. This latter step can take several minutes. It is completed when three white lines appear on the screen (which are the lines that were drawn in step 1.12). Press "Save container info" button.
17. Push the 'Collect volume' button. Now, as shown in Figure 6, consecutively mark the 8 corners of the water body, which is a cuboid. For every point first click it in the front view and then in the top view.
18. Select '3D' on the left corner of the window. Now a new window opens showing the outline of a box with a white bottom (as in Figure 15, but without the trajectory). Right click the mouse to select rotations and translations of the box view. If no image appears or if the image is not that of a cuboid (i.e. in step 1.17 the corners were right-clicked in the wrong order), do the following: while leaving, but not closing the Trajectory Module, using Windows go to the calibration folder (made in step 1.6), select subfolder 'Animal container data' and delete the file named TankVertices and repeat step 1.17.
19. Congratulation, the calibration is completed. Ideally, a test run with a dummy fish should be performed to check if everything works well. The good news is that recalibration has only to be performed if the positions of mirror or camera were (even only slightly) altered or if there are problems with tracking (see section 3).

2. Recording

1. Place the observation container filled up to the marks with water into the observation chamber such that the feet of the container are inserted into the holes provided for correct alignment. Orient the container such that the semi-transparent walls (or white walls in our example – see Discussion) face the far side and to the right as seen from the camera.
2. Switch on the lights in the compartment, put the (experimental or dummy) fish into the container and close the curtains. Note: the system can record multiple fish (see the Representative Results section); however, it is advisable to gain some experience with the system by recording single fish first.
3. Open the Record Module. On the screen appears the image of the front and top (mirror) view of the container. The image might not be ideal (e.g. overexposed, e.g. Figure 7) and might have to be adjusted (next three steps) to obtain good contrast between fish and background (e.g. Figure 8).
4. Select 'Camera control' and, in the 'General Settings' window that opens, adjust 'Shutter' and 'Gain' (after deselecting the corresponding boxes to the right, see Figure 9) in such a way that the image is not overexposed and there is good contrast between fish and background.
5. On the left side of the 'Camera control' window select 'White Balance/Color' and in the new window deselect the 'Auto' box and adjust the Red and Blue values (see Figure 10) to obtain best color contrast in both front and top view.
6. It might be necessary to switch a few times between adjusting color (under 'White Balance/Color') and adjusting gain and shutter (under 'General Settings') to obtain the best camera setting. Be aware that even if you were to be satisfied without any adjustment, it is crucial to deselect the 'Shutter', 'Gain' and 'Auto' boxes (as explained in steps 2.4 and 2.5), otherwise the background will fluctuate over the course of the recording which makes it impossible to extract the trajectories. Do not change any of the other settings. When done close the 'Camera Control' window.
7. From the main menu of the Record Module select 'Data location' to make or to open an earlier made folder. Note that every recording (e.g. fish or shoal) gets its own folder which should ideally be a subfolder of the experiment folder.
8. Press 'Start recording'. A window opens that asks for the duration of the recording in minutes (only enter whole numbers). Press 'OK', which starts the actual recording session.
9. If the recording is completed, exit the program. Before recording the next fish it is recommended to restart the computer. Use the Restart option in Windows to preserve the camera settings (from steps 2.4 and 2.5). The necessity to restart the computer between recordings is due to some problems with the camera card.
10. Before going to the Trajectory Module for path reconstruction, copy and paste the 'Animal container data' and the 'Correspondence data' subfolders from the calibration folder (made in section one of this protocol) to every folder that was created during recording. In these latter folders two other subfolders are present, namely 'Raw data' and 'Time stamps'. The former contains all recorded frames in JPEG format; the latter contains information about the total number of frames and the intervals between the frames (in msec).

3. Path Reconstruction

1. Open the Trajectory Module, press 'Data location' to select the folder that was created with the Record Module and then press the 'System Setup' button to open the 'Parameters' (see Figure 11).
2. Select a Top and Bottom View Color Thresholds. A good starting value is 1,000. If this value is low, the software is more likely to record noise. However, if the value is too high, the fish might be below detection range. Later (steps 3.11 and 3.12) it might become necessary to readjust the color thresholds. Also note that the top and bottom color thresholds have not to be the same.
3. Select Top and Bottom View Size Thresholds. A good starting value for a midsized zebrafish is 10, for a very big one 20, and for a very small one 5. The higher the size threshold, the less likely it is to pick up noise. However, if it is too high, the software might not be able to detect the fish either. Again, in step 3.11 and 3.12 it might become necessary to readjust the size thresholds. Also note that the top and bottom size thresholds have not to be the same.
4. For 'Upper Size Threshold' a value of 3,000 is usually good and for 'Specific Length' leave it at 220 (already preset). The 'Number of fish' refers to the number of fish in the observation tank. 'Max num of Points' refers to the total number of frames. Remember (or write down) this number or a slightly lower one (e.g. 45,703 might be memorized as 45,000). Press OK.
5. Sequentially press the buttons 'Load correspondence w/o water', 'Calibrate virtual camera(s)' and 'Load animal container info'. An error message will be generated if the calibration-subfolders 'Animal container data' and 'Corresponding data' were not copied to the current experimental folder (see step 2.10). If so, copy and paste those subfolders now and press the buttons again.
6. Press the button 'Acquire background'. A new window will ask to fill in 'from' and 'to' with the default values 0 and 100, respectively. Change the 100 into the maximal number of frames (in the example given in step 3.4 this might be simplified as 45,000). Press OK.
7. Wait until a (distorted) image appears. Then press 'Load background' which will result in establishing the correct background image. The fish should not be visible in this background image. If the fish is visible, try to acquire a better background from other recording segments (e.g. try 0-1,000, 1,000-2,000 etc. in step 3.6).
8. If the fish did not move at all during the recording session, the only way to acquire a background is to remove the fish and make a new short recording without the fish, but with the same 'Camera Controls' settings (this is important). Acquire the background and copy the 'Background data' subfolder to the current experimental folder. Now press 'Load background', do not press 'Acquire background'. Note that opening and closing the curtain often results in somewhat changed background, thus it might become necessary to adjust the thresholds in steps 3.2 and 3.3 to reduce noise. Step 3.11 shows how to check for noise.
9. After loading the background, adjust the three red boxes (with the help of the mouse) as depicted in Figure 12. The boxes mark the areas in which the software is looking for the fish. Close the box on the left (i.e. reduce it to a line or a point) because the side view is only used for calibration (as was explained in step 1.7).
10. Press 'Start generating trajectories' button. A window opens asking for the starting frame number. Go with the default '0'. After pressing 'OK', the 'Tag assignments' window opens (see Figure 13). Press 'Step forward' until the tag ('X0') appears on the screen. If the tag is not placed on the image of the fish, press 'No' until it is placed correctly. Then press 'Go'. If it is never assigned correctly (or at all) press 'Go' anyway and look for the problem (next step).
11. Path reconstruction starts. Ideally, the tag should always (or at least most of the time) be placed on the image of the fish. If it is often placed wrongly, open the 'Info' window (second choice above the video screen). Figure 14A shows the ideal case, where only the outlines of the fish are seen. Figure 14B shows a case of excessive noise which prevents the correct tag assignment. If you see a clear image of the fish without noise (as in Figure 14A), but the tag will never or only seldom be assigned correctly, then the most likely reason is that the calibration is off. Recalibrate and redo the path reconstruction. If during calibration, mirror and cameras are not moved, the new calibration data can be used for the previously performed recordings.
12. In case of excessive noise (as in Figure 14B), press 'Stop' in main menu and choose new color and size thresholds (steps 3.2 and 3.3). Also check if the red boxes (step 3.9) can be reduced a little bit (for instance to exclude the noise generated by water movements at the top). Note that sometimes the program is 'stuck' (if no tag could be allocated at all): in this case restart the program.
13. Watch the trajectory on the 3D-screen (third choice above the screen). Figure 15. represents an example of a trajectory. By right-clicking the mouse select between rotation and translations of the 3D-configuration.
14. When the trajectory is completed, sequentially press the buttons 'Save' (not 'Stop', which deletes the trajectory!), 'Trajectory clean-up' and 'Trajectory smoothing'. A new window opens showing the filter-parameters for smoothing (Figure 16). Do not alter the default values before having gathered more experience with the system. Press 'OK'.
15. By now it should be clear why choosing a good camera setting (with good contrast between fish and background) before recording (see steps 2.4-2.6) was so important. Proceed to Data Processing.

4. Data Processing

1. First, take a look at the folder that was used for recording the fish (Figure 17). It contains 'Animal container data' and 'Correspondence data' subfolders which were copied from the calibration folder (step 2.10), the 'Time stamps' and 'Raw data' subfolders which were created during recording of the fish and the 'Background data' and 'Trajectories' subfolders which were created during path reconstruction.
2. Figure 17 also shows that the 'Trajectories' subfolder contains three files. The point.smoothed-file is the file that will be used for further data processing. In case of multiple fish recording (e.g. shoaling experiment), for every fish a separate point.smoothed-file will be generated.
3. Figure 18 shows the top part of an example point.smoothed-file. The first line gives the total number of samples or frames, the second line the duration of the recording (in minutes) and the third line the coordinates (sequence: x, z, y) of the center of the water body relative to the camera. This is followed by one line for every frame giving the x-, y-, z-coordinates (in mm) for the fish and the duration of that sample (in seconds).
4. There are two ways to process the data provided in the point.smoothed-files: use the Data Processing Module or use other software, such as Excel or MATLAB to do calculations which are not (yet) incorporated in the Data Processing Module.
5. Open Data Processing Module, select 'Project' on the toolbar, then from the pull-down menu select 'New' and then fill in a name for the experiment. Select 'Edit Project' on toolbar, select 'Add to Project' from the pull-down menu, followed by 'New group' and fill in a name for the group (e.g. 'Controls'). Note, at this moment the project and group names are not really used, but the names have to be filled. The created folders could be used to store the output files (see step 4.10).
6. Select a group (e.g. 'Controls') in the right window (see Figure 19). The tool bar will change. Select 'Edit group', then 'Add to group' from the pull-down menu, then 'New measurements'. This will open a browser. Open the folder made during recording (step 2.7), select the subfolder 'trajectories' and then the point.smoothed-file. In the example presented in Figure 19, for Group-A six files were loaded, for Group-B four files. (Note that in the illustrated example the smoothed files have the extensions 0 and 1, because in that experiment two goldfish were tested simultaneously).
7. For analysis, select the point-smoothed-file in the right window, select 'Data Processing', select 'Single Endpoints' from the pull-down menu and by way of example select 'Duration freezing' (Figure 20). A window opens to specify freezing in terms of speed threshold and duration threshold. In the example the fish spends 25.86 % of the observation time on freezing.
8. To familiarize yourself within the program, try other examples, such as Travel distance, Depth Distribution (which allows you to determine the number of depth levels) and Burst frequency.
9. There are more advanced calculations, such as spectral analysis of for instance the z-component of acceleration. In this case, select 'Time-series features', then 'TD freq domain analysis', 'Spectral analysis' and 'Z accele spectrum'. Fourier analysis will be performed on the indicated parameter (for explanation⁷). Not all options displayed on the menu are yet fully implemented. For instance, 'Path tortuosity' was only added experimentally.
10. When selecting the option 'Batch processing using default threshold', about 200 endpoints will be calculated. In this context, 'default' means that delimiters have been set. For example, currently depth distribution is calculated for three depth levels, angular distribution is calculated for four sections (i.e. quadrants) etc. After generating the parameters a new window opens that asks for the location and name of the file to be created. Those will be saved in CSV-format and can be imported into other software, such as Excel, for further processing.
11. In contrast to the Recording Module and Trajectory Module, the Data Processing Module is an evolving module that will be further adapted to the requirements of the researchers.