PhotoT iming: Triggering Video Capture


Photo timing of finishers in an athletics running event can be facilitated by recording a video of the finish and processing the video frames. The frames can be individually inspected or stitched together to create a linear image. As each frame occurs for a fixed time, eg 1/30th second they can be used to measure finish time to 1/10th second or better. The issue is how to get an absolute time reference on the video for the event start.

Nb The camera at hand, a Google Pixel 6, can record at 60 FPS facilitating an accuracy of better than 20 mS. Apple iPhones 14 Pro Series (240fps).

Context

  • Can capture Video with Audio on an Android phone.
  • Objective is to use the video, taken across the finish line at an athletics event, for timing and finish positions of competitors.
  • Video preview has crosshairs overlay to assist phone positioning wrt the finish line.
  • Can stitch middle line of pixels from successive video frames into an image with time market overlays, into a continuous image.
  • Can place a cursor on that image to determine the position and time of athletes crossing a line.

Issue at hand

How to trigger the start of the video capture and to accurately record the event starts time.

Discussion

The first and simplest solution is to manually start the video when the start gun fires by pressing a button in the app’s UI. This will be the least accurate method and will be subject to any latency with starting the recording both ap and human. It could be improved by having the video capture software actually progress through all of the finally start code up to where it actually calls video start and do a busy wait Task.Wait(1) … until signalled (manually on this case) to start capture. Upon inspection there are a couple of lines of code from when the start button is pressed to when

_mediaRecorder.Start();

is called. Either way it will be subject to human limitations. Typically, human responses have a limitation of at least 0.1 of a second:

‘The IAAF has a controversial rule that if an athlete moves in less than 100 ms, it counts as a false start, and since 2009, they must be disqualified – even despite an IAAF-commissioned study in 2009 that indicated top sprinters are able to sometimes react in 80–85 ms.[45]” Wikipedia

An improvement would be where the video capture, already in this busy wait state is automatically triggered by the start gun. mediaRecorder. If not an umbilical chord to the gun, then the signal could triggered by a flash from the gun or from its explosive sound. The signal would then be relayed by WiFi, Bluetooth or a wired connection. Without a physical connection to the phone, a combination of the second and third or even the first and second methods could be used. For these, the Bluetooth connectivity would be integrally to the video capture app. Upon inspection there are only a few lines of code from Start() to when the recording will actually be started. The system could be calibrated by measuring this latency.

Another approach suitable for short races such as sprints would be to have the video capture started just prior to the race start. There would then be some sort of overlay to the video image at race start such as a flash signal in the forward view of the phone’s camera that is triggered by the start gun. Finish times are then relative to that start signal, not the video start. Alternatively the start gun’s sound could be captured by the video’s audio (hence the addition of the audio capture to the library) with parallel analysis of the video frames and the audio signal. A third approach would be that a signal from the start gun is able to modify the video stream, such as one frame to be blacked out.

Intercepting the video stream while using Camera2 with MediaRecorder requires accessing the raw frames before they are encoded. Since MediaRecorder directly handles encoding and writing to a file, it doesn’t provide direct access to individual frames.
GitHub Copilot

The GitHub Copilot suggestions are:

  • Using ImageReader (Intercept Frames Before Encoding). Instead of sending frames directly to MediaRecorder, you can attach an ImageReader to the camera and process frames before they are recorded.
  • Using MediaCodec (Intercept Encoded Stream). If you need access to the encoded video stream, you can use MediaCodec instead of MediaRecorder.

Which Approach Should You Use?

  • Use ImageReader if you need raw frames before encoding.
  • Use MediaCodec if you need encoded frames before writing to a file.

Sounds like the ImageReader approach would be best in this, replacing a raw frame with a black from or some other start signal frame.

2DO …

 TopicSubtopic
   
 This Category Links 
Category:Xamarin Index:Xamarin
<  Prev:   MAUI-Android