MuHAVi: Multicamera Human Action Video Data

including selected action sequences with

MAS: Manually Annotated Silhouette Data

MuHAVi-uncut: Full videos with realistic Silhoutte Data

for the evaluation of human action recognition methods

Last updated September 2017

Originally part of the REASON project funded by the UK's Engineering and Physical Sciences Research Council (EPSRC)

Then part of the OBSERVE project, funded by the Fondecyt Regular Program of the Chilean Research Council for Science and Technology (Conicyt), grant number 1140209.

Currently part of the UC3M-Conex project CloseVU (Close View Visual Understanding) project, funded by the Marie Curie EU Programme.

This dataset was originally put together by a team based at Kingston University's then  Digital Imaging Research Centre and continued by a team at the Department of Informatic Engineering at the University of Santiago de Chile.

The updates in Feb-July 2014  and from Sept 2015 have been done as part of a Chair of Excellence/Marie Curie Professorship stay by Prof. Sergio A Velastin at the Applied Artificial Intelligence Group of the Universidad Carlos III de Madrid

Organizing the experiments and data collection were performed by the project's team based at Kingston University together with the partners based at Reading University and  UCL's Pamela Laboratory.

NOTE: We have also made our virtual human action silhouette data available online (visit our ViHASi page).

If you publish work that uses this dataset, please use the following references:

For MuHAVi-uncut:

@article{murtaza2016multi,
  title={Multi-view human action recognition using 2D motion templates based on MHIs and their HOG description},
  author={Murtaza, Fiza and Yousaf, Muhammad Haroon and Velastin, Sergio A},
  journal={IET Computer Vision},
  volume={10},
  number={7},
  pages={758--767},
  year={2016},
  publisher={IET Digital Library}
}

DOI 10.1049/iet-cvi.2015.0416

For MuHAVi-MAS:

@inproceedings{singh2010muhavi,
  title={Muhavi: A multicamera human action video dataset for the evaluation of action recognition methods},
  author={Singh, Sanchit and Velastin, Sergio A and Ragheb, Hossein},
  booktitle={Advanced Video and Signal Based Surveillance (AVSS), 2010 Seventh IEEE International Conference on},
  pages={48--55},
  year={2010},
  organization={IEEE}
}

DOI: 10.1109/AVSS.2010.63

You can also find here a list of publications that use the MuHAVi dataset.

New (12.09.2017):

• A brand new set of temporal ground truths for MuHAVi-uncut has been prepared that defines start-end of each "sub-action" in an action
• Silhouttes for each of the sub actions in MuHAVi-uncut are now available 
  

Introduction

We have collected a large body of human action video (MuHAVi) data using 8 cameras. There are 17 action classes as listed in Table 2 performed by 14 actors. We initially processed videos corresponding to 7 actors in order to split the actions and provide the JPG image frames. These include included some image frames before and after the actual action, for the purpose of background subtraction, tracking, etc. The longest pre-action frames correspond to the actor called Person1. Note that what we provide is therefore temporally pre-segmented actions as this was typical when the dataset was first released. We now  (see below) provide long unsegmented sequences for people to work on temporal segmentation.

Each actor performs each action several times in the action zone highlighted using white tapes on the scene floor. As actors were amateurs, the leader had to interrupt the actors in some cases and ask them to redo the action for consistency. As shown in Fig. 1 and Table 1, we have used 8 CCTV Schwan cameras located at 4 sides and 4 corners of a rectangular platform. Note that these cameras are not synchronised. Camera calibration information may be included here in the future. Meanwhile, one can use the patterns on the scene floor to calibrate the cameras of interest.

Note that to prepare training data for action recognition methods, each of our action classes may be broken into at least two primitive actions. For instance, the action "WalkTurnBack" consist of walk and turn back primitive actions. Further, although it is not quite natural to have a collapse action due to shotgun followed by standing up action, one can simply split them into two separate action classes.

We make the data available to the researchers in computer vision community through a password protected server at the University Carlos III de Madrid, Spain. The data may be accessed by sending an Email (subjected "MuHAVi-MAS Data") to Prof Sergio A Velastin at sergio.velastin@ieee.org giving the names, email addresses and institution(s) of the researchers who wish to use the data and their main purposes. We request this only to build a list of people using this dataset to form a "MuHAVi community" with whom to communitcate. The only requirement for using the MuHAVi data is to refer to this site and to our publication(s) in the corresponding publications.

Figure 1. The top view of the configuration of 8 cameras used to capture the actions in the blue action zone (which is marked with white tapes on the scene floor).

Table 1. Camera view names appearing in the MuHAVi data folders and the corresponding symbols used in Fig. 1.

On the table below, you can click on the links to download the data (JPG images) for the corresponding action

Important: We noted that some earlier versions of that earlier versions of MS Internet Explorer could not download files over 2GB size, so we recomment to use alternative browsers such as Firefox or Chrome.

Each tar file contains 7 folders corresponding to 7 actors (Person1 to Person7) each of which contains 8 folders corresponding to 8 cameras (Camera_1 to Camera_8). Image frames corresponding to every combination of action/actor/camera are named with image frame numbers starting from 00000001.jpg for simplicity. The video frame rate is 25 frames per second and the resolution of image frames (except for Camera_8) is 720 x 576 Pixels (columns x rows). The image resolution is 704 x 576 for Camera_8.

Table 2. Action class names appearing in the MuHAVi data folders and the corresponding symbols used in Fig. 3.

Table 3. Actor names appearing in the MuHAVi data folders and the corresponding symbols used in Fig. 3.

*** end of historical note

So far, MUHaVi has consisted of

• (Temporally) manually pre-segmented action sequences (in JPEG files)
• Manually annotated silhouettes for a small sub-set of actors/actions/cameras (MuHAVi-MAS dataset).
  

Thanks to work done by Dr Zezhi Chen of Kingston University, Jorge Sepúlveda of the University of Santiago de Chile (USACH) and Prof. Sergio A Velastin from the Universidad Carlos III de Madrid, we are now able to provide:

• Un-cut original video sequences (mainly in MPEG2) for each camera (the recordings are continuous and contain the acted actions but also the gaps and breaks in between). 
• Ground truth describing times of start and completion (frame numbers) of each sub-action in each video file by each actor (Note: the community's views are wellcome to agree on a set of metrics to evaluate temporal segmentation methods)
• Silhouettes computed by Z.Chen´s algorithm (the rationale is that these are realistic silhouettes typical of the state of the art and people are invited to test the robustness of their human action recognition and temporal segmentation algorithms based such realistic, and "imperfect", segmentation)

Each compressed archive contains files named %d.png where the number is the frame number. In each, file black (0) represents the background, white (255) the foreground i.e. the silhouette and grey (127) is a detected shadow (normally to be considered as background).

The "Parameter" is one of the factors that affect foreground detection in terms of true positives vs false positives. When tested against the manually annotated silhouettes, a value of 3.0 produces a TPR (true positives rate) of around 0.78 and a FPR (false positives rate) of around 0.027 while 4.0 gives around 0.71 and 0.013 and 5.0 gives 0.625 and less than 0.01 (i.e. less noise but less foreground). As many of the false positives tend to be noise outside the main silhouette, we expect that most people will use the set with higher TPR and reduce the false positives e.g. with morphologial filtering. When publishing results can you please ensure that you give full details of any pre-processing of this kind.

• Each action (e.g. "walk and turn back") was conducted by each actor a number of times (typically 3), but the annotation only contained the start and end of the (3) actions as a whole and not of each one separately.
• Actions such as "walk and turn back" could really regarded as two or three sub-actions: walk (toward one end of the stage), turn, walk (back to the other end of the stage) and it would be nice to annotate them separately
• Finally, we found that there were errors in the annotation!
  

1. The first column refers to the camera and actor numbers.
2. The second column header gives the action (e.g. "WalkTurnBack", "RunStop").
3. The numbers on the second column for each person give the frame number, in the video sequence, where the action starts and in the third column where it ends (this is somewhat subjective, of course and the community needs to agree on a metric that would not unjustly penalise algorithms).
4. If the action was repeated (that is almost always the case) the start and end frames are given in the fourth and fifth columns and so on

Note: this material is only historical (as we do not have well documented sources of these results)

Masks obtained by applying two different Tracking/Background Subtraction Methods to some of our Composite Actions

Each zip file contains masks (in their bounding boxes) corresponding to several sequences of composite actions performed by the actor A1 and captured from two camera views (V3 and V4) for the purpose of testing silhouette-based action recognition methods against more realistic input data (in conjunction with our MAS training data provided below), where the need for a temporal segmentation method is also clear.

More data and information to be added ...

*** end of historical section

MuHAVi-MAS: Manually Annotated Silhouette Data

We recommend using this subset of MuHAVi to test Human Action Recognition (HAR) algorithms independently of the quality of silhouettes. For a fuller evalution of a HAR algorithm, please consider using MuHAVi "uncut" instead.

We have selected 5 action classes and manually annotated the corresponding image frames to generate the corresponding silhouettes of the actors. These actions are listed in Table 4. It can be seen that we have only selected 2 actors and 2 camera views for these 5 actions. The silhouettes images are in PNG format and each action combination can be downloaded as a small zip file (between 1 to 3 MB). We have also added 3 constant characters "GT-" to the beginning of every original image name to label them as ground truth images.

On the table below, you can click on the links to download the silhouette data for the corresponding action combinations.

Table 4. Action combinations corresponding to the MAS data for which ground truth silhouettes have been generated.

NEW! The table below contains links to the corresponding AVI video files (in MPEG2) from which the JPEG file sequences were extracted which were then used by the manual annotators to get the silhouettes (Note that due to a software bug the JPEG sequences had a couple of frames missing towards the end of the sequence, therefore the AVI files would not exactly correspond to the silhouette frames. As this happens toward the end, it should not significantly affect work that evaluates automatic silhouette segmentation and that uses performance metrics based on aggregating an averaging results over the whole sequence)

Table 5. Action combinations corresponding to the MAS data for which ground truth silhouettes have been generated.

Finally, the following table documents the frames that were manually segmented so that you can test foreground segmentation algorithms (i.e. this table tells you the correspondence between JPEG, AVI and PNG frames in the dataset). Please note that the human annotators worked on the JPEG files and hence there is a one to one correspondence between JPEG and PNG files. Because of a bug we later discovered on the version of mplayer that was used to generate the JPEG frames, there is small difference in the number of frames in the AVI files, but we still suggest you use the AVI files as the JPEG were effectively transcoded from the original MPEG2 videos)

We have reorganized these 5 composite action classes as 14 primitive action classes as shown in the table below.

You may download the data by clicking here (32MB).

These 14 primitive action classes may also be reorganized in 8 classes where similar actions make a single class as shown in the table below.

Figure 2. Sample images of annotated silhouettes from the MAS data (for actor A1) corresponding to 20 selected action sequences (5 action classes, 2 actors and 2 cameras) from the MuHAVi data (as listed in Table 4).

Figure 3. Sample image frames from the MuHAVi data for 17 action classes, 7 actors and 8 camera views (as listed in Table 1, 2 and 3, and, shown in Fig. 1).

Publications that use the MuHAVi dataset (if you have any not listed here please let me know):

Singh, Sanchit, Sergio A. Velastin, and Hossein Ragheb. "Muhavi: A multicamera human action video dataset for the evaluation of action recognition methods." In Advanced Video and Signal Based Surveillance (AVSS), 2010 Seventh IEEE International Conference on, pp. 48-55. IEEE, 2010.

Marlon Alcântara, Thierry Moreira, and Hélio Pedrini, “Real-time action recognition based on cumulative motion shapes,” in Acoustics, Speech and Signal Processing (ICASSP), 2014.

@inproceedings{alcantara2014,
  author = {Alc\^antara, Marlon and Moreira, Thierry and Pedrini, H\'elio},
  title = {Real-Time Action Recognition Based On Cumulative Motion Shapes},
  booktitle = {Acoustics, Speech and Signal Processing (ICASSP)},
  year = {2014},
}

A. A. Chaaraoui, P. Climent-Pérez, and F. Flórez-Revuelta, “A review on vision techniques applied to Human Behaviour Analysis for Ambient-Assisted Living,” Expert Systems with Applications, vol. 39, no. 12, pp. 10873–10888, 2012.
Available at ScienceDirect: http://www.sciencedirect.com/science/article/pii/S0957417412004757

-          Climent-Pérez, Pau, Alexandros Andre Chaaraoui, and Francisco Flórez-Revuelta. "Useful Research Tools for Human Behaviour Understanding in the Context of Ambient Assisted Living." In Ambient Intelligence-Software and Applications, pp. 201-205. Springer Berlin Heidelberg, 2012.
Available at SpringerLink: http://link.springer.com/chapter/10.1007%2F978-3-642-28783-1_25?LI=true
Also uploaded at ResearchGate: http://www.researchgate.net/publication/224960636_Useful_Research_Tools_for_Human_Behaviour_Understanding_in_the_Context_of_Ambient_Assisted_Living