Abstract
NCTU-GTAV360 is new 360° action recogntion video dataset captured from Grand Theft Auto V (GTA V) game with RiotMode enabled. It will gave the pedestrians in the game have weapons and attack each other. Despite many action recognition video dataset available right now, none of them are in spherical projection. The spherical video is obtained by stitching views from various angles, combined them into a video. Script Hook V and Script Hook V .NET was used to control the camera rotation and position automatically. The benefit of using 360° camera is it can capture more angles than the normal one. The spherical video captured in 200 locations within the Los Santos city (the city name in the GTA V). We believe this dataset will also help other research in spherical domain, especially on action recognition.
Video Samples
Note: the videos shown above may be compressed by YouTube. You can get the full resolution of video above here:
Resources
- Paper:
S. Ardianto, and H-M. Hang. "NCTU-GTAV360: A 360° Action Recognition Video Dataset". IEEE 21st International Workshop on Multimedia Signal Processing. 2019. - Data:
- 200 locations
Please send an email request to sandyardianto.03g@g2.nctu.edu.tw to download the dataset (videos + labels)
- 200 locations
Data Format
Video Files:
Label Files:
The videos are stored in mp4 at 20 FPS using ffmpeg software. File name format is <location number>.mp4. For example 123.mp4 is the video for location 123.
The video resolution is 4096x2048.
The video resolution is 4096x2048.
Label Files:
- Pedestrian:
Each frame will have one txt file consist n row lines. Which n is the number of nearby pedestrians. File name format is <frame number>.txt.
Every line contains name of the action and 26 x 3 (x,y,z) skeleton joint location in meter.
Example:
Fleeing 78.72278 26.1908 0.6010818 78.66736 26.16058 0.5071983 78.55249 26.11688 0.08034134 78.54089 26.10248 -0.02093887 78.58105 26.13025 0.2761879 78.68774 26.17072 0.4654961 78.58105 26.13025 0.2761879 78.59106 26.35889 0.4388542 78.46826 26.41992 0.15522 78.75415 26.41547 0.1192818 78.77661 26.39716 0.1379128 78.81555 26.35748 0.146553 78.84436 26.33582 0.1467934 78.74158 25.96515 0.4465485 78.57336 25.73566 0.3207016 78.72839 25.79449 0.09486008 78.75525 25.8125 0.08260345 78.78296 25.84119 0.04288483 78.80005 25.85321 0.0136261 78.48596 26.17224 -0.1030998 78.34131 26.00775 -0.4455719 78.01257 25.75787 -0.4383545 77.96021 25.69379 -0.5770416 78.58215 26.00708 -0.08285522 78.82849 26.16333 -0.3659515 78.82947 26.22046 -0.7749634
Running -41.27747 -36.43158 -22.17011 -41.25598 -36.45203 -22.28863 -41.23596 -36.46753 -22.70668 -41.24634 -36.46429 -22.77118 -41.24829 -36.45447 -22.53161 -41.29114 -36.43036 -22.32133 -41.24829 -36.45447 -22.53161 -41.35803 -36.58032 -22.36565 -41.34436 -36.67072 -22.63333 -41.50012 -36.58838 -22.80621 -41.51758 -36.59198 -22.81009 -41.55774 -36.60358 -22.83668 -41.59277 -36.60132 -22.86186 -41.16589 -36.31537 -22.32129 -41.18188 -36.10687 -22.53466 -41.37317 -36.2724 -22.63134 -41.3844 -36.28125 -22.61987 -41.39966 -36.32819 -22.62388 -41.41809 -36.36261 -22.64219 -41.28174 -36.55688 -22.83057 -41.276 -36.56982 -23.26937 -41.17969 -36.61194 -23.65543 -41.30688 -36.58838 -23.7084 -41.19495 -36.38867 -22.84185 -41.3324 -36.32819 -23.25438 -41.18396 -36.45209 -23.60468 - Vehicle
Each frame will have one txt file consist n row lines. Which n is the number of nearby vehicles. File name format is <frame number>.txt.
Every line contains type of the vehicle, position (x,y,z), and its dimension (width, length, height) in meter.
Example:
Car 581.0228 39.24371 92.82274 2.010509 5.55837 1.650319
Bike 502.468 19.51715 88.9951 2.000296 4.716197 1.666888 - Weather
Each locations will have one weather file stored in txt which only contains one line of what weather showing on that location. File name format is <location number>.txt.
Example: Overcast
Capture Method
Capture 24 views
Since the game did not support to capture 3600 images, so we built a program to move the camera to look at various angles. The camera rotation range in the game is -600 to 600 for X-axis, and -1800 to 1800 for Z-axis. Y-axis only used for direction instead of rotation. In this case, we used 400 interval for X-axis, and 600 interval for Z-axis. The camera cannot be moved when the game is paused. Hence, we need to slow down the game time and continuously pause then play the game again while moving the camera to capture the scenes.
Stitch into equirectangular projection
We used PTGui Pro software to stitch the 24 images into equirectangular projection. There are many spherical projections available right now, however, this projection is the most popular one in representing the 360° image or video. Many companies, like Facebook, YouTube, GoPro, Ricoh, are using this projection in their product. Figure 3 shows the position of the images when they are in the equirectangular projection.
Images from previous step stitched into equirectangular projection with its guideline:
Details on each image warped into equirectangular projection:
No Image Warped Yaw Pitch Roll No Image Warped Yaw Pitch Roll No Image Warped Yaw Pitch Roll 0 
0 -60 -0.06 8 
-119.9 -19.9 -0.08 16 
120.1 19.9 0.0005 1 
-59.9 -59.9 -0.02 9 
-179.9 -20 -0.1 17 
60.1 19.9 0.08 2 
-119.8 -59.9 0.2 10 
120.1 -20.1 -0.07 18 
0.3 60 0.1 3 
-179.6 -60 -0.3 11 
60.1 -20 0.03 19 
-59.9 60 0.09 4 
120.2 -60.1 -0.2 12 
0.1 20 0.1 20 
-119.8 60.1 0.2 5 
60.1 -60.1 -0.03 13 
-59.9 20.1 0.05 21 
-180 60.1 -0.1 6 
0.07 -20 0.06 14 
-119.9 20.1 0.006 22 
120 59.9 -0.2 7 
-59.9 -19.9 0.02 15 
-179.9 20 -0.05 23 
60.3 60 0.4 Combine into video
There are many video encoding format, such as mp4, mkv, avi, etc. The images from previous steps are stored in jpg format. We choose the mp4 to store the images into video. FFmpeg software used to combine the images into video. The video result will have same resolution with equirectangular projection images that is 4096x2048.
Dataset Statistics
Weather statistics
Weather # location Percentage (%) Overcast 87 43.5 Smog 56 28 ExtraSunny 43 21.5 Clouds 12 6 Raining 2 1 Vehicle statistics
Type Number of vehicle Percentage (%) Car 15,472,691 97.81 Bike 143,256 0.91 Bicycle 107,384 0.68 Helicopter 39,135 0.25 Quadbike 26,777 0.17 Boat 20,683 0.13 Vehicle 5,028 0.03 Train 4,481 0.03 Actions statistics
Action Close (<5m) Near (5m-20m) Far (>20m) Full # % # % # % # % OnFoot 35,810 39.50 459,858 27.44 13,778,508 49.53 14,274,176 48.25 AimingFromCover 8,563 9.44 475,618 28.38 4,790,407 17.22 5,274,588 17.83 Prone 10,802 11.91 183,809 10.97 3,154,112 11.34 3,348,723 11.32 Running 4,458 4.92 67,596 4.03 2,225,398 8.00 2,297,452 7.77 Walking 8,845 9.76 67,865 4.05 1,479,794 5.32 1,556,504 5.26 Fleeing 1,005 1.11 81,203 4.84 731,302 2.63 813,510 2.75 Stopped 19,978 22.04 296,497 17.69 718,589 2.58 1,035,064 3.50 Swimming 0 0.00 0 0.00 412,305 1.48 412,305 1.39 GoingIntoCover 1,006 1.11 22,210 1.33 250,953 0.90 274,169 0.93 Falling 0 0.00 81 0.00 66,347 0.24 66,428 0.22 Unknown 177 0.20 17,517 1.05 58,851 0.21 76,545 0.26 Ragdoll 0 0.00 3,164 0.19 45,590 0.16 48,754 0.16 Climbing 0 0.00 285 0.02 45,033 0.16 45,318 0.15 BeingStunned 0 0.00 0 0.00 25,899 0.09 25,899 0.09 GettingUp 0 0.00 188 0.01 11,221 0.04 11,409 0.04 OnBike 0 0.00 20 0.00 10,997 0.04 11,017 0.04 SwimmingUnderWater 0 0.00 0 0.00 4,654 0.02 4,654 0.02 Reloading 20 0.02 90 0.01 3,844 0.01 3,954 0.01 Vaulting 0 0.00 92 0.01 3,122 0.01 3,214 0.01 GettingIntoAVehicle 0 0.00 0 0.00 531 0.00 531 0.00 Jumping 0 0.00 0 0.00 56 0.00 56 0.00 DoingDriveBy 0 0.00 0 0.00 1 0.00 1 0.00