GitHub Copilot v Documentation: How far can you go with Copilot


With clear and concise specifications to GitHub Copilot for an app to be coded, how far into the app development can you go with Copilot. Is there a limit? An app with variations was developed as a Console then WPF ap with Copilot directing how to code it using the OpenCVSharp image processing library. When it came to using the same library with MAUI as Windows phone, … MAUI was OK on the dektop with a suitable runtime being available. But on the phone there was not a runtime packaged with that library so Copilot was tasked with porting the library to use the Engu.CV library. That worked to a point but the documentation for Managed Engu.CV was required to be consulted for some technical issues.

About the App

The app is to stitch together a phototiming image from a video. I racing sports such as athletics, cycling and horse racing, a high speed video it taken across the finsih line and a phototiming image is craeted by taking the middle b=vertical line of each video from and sequentially appending on the right of the iamge so far. The iamge can then be exampled to see when competitors first “cross the finsih line”.

The objective of this app is to use a standard video instead of the high speed camera. It can be recorded using a phone. So there are only 30 frames per second so resolution is about 0.03 of a second.

I have written an app that does this some year sago with an off-the-shelf image procesing library but that won’t build or run now. Time to build one that will run on a Windows Deskop or a Phone.

The devloped app is in three versions which are indication of the app development:

  • A Console app that does the curve stitching from the video
  • A WPF that does the curve stitching, displays it and can used to determine times by placing the mouse cursor at positions, horizontally, on the image
  • A MAUI app that does the curve stitching directly from a video on the phone and displays it (ONLY).

The apps all use one of two libraries contained in the rpeository to do the curve stitching and to add timining marks at the bottom of the image.

Both of these last two links are .NET wrappers for the OpenCV computer vision library.

Getting started … and more

GitHub Copilot was tasked with getting styarted with the Console app, Started with the following request to Copilot:

“Extract the middle vertical pixels from a video frame”
then
“In C# only”

Response:

Certainly! To extract the middle vertical pixels from a video frame in C#, you can use the OpenCV library through the OpenCvSharp wrapper

And so it began

“and stitch pixels vertical lines into an image”

Response

To stitch vertical pixel lines into an image, you can collect multiple vertical lines (like the middle vertical line of frames in a video) and merge them horizontally into a single image. Here’s how you can do it in C# using the OpenCvSharp library:

Install-Package OpenCvSharp4

using OpenCvSharp;
using System;
using System.Collections.Generic;

class Program
{
    static void Main(string[] args)
    {
        // Path to the video file
        string videoPath = "your_video.mp4";

        // Open the video file
        using var capture = new VideoCapture(videoPath);
        if (!capture.IsOpened())
        {
            Console.WriteLine("Failed to open the video file.");
            return;
        }

        // List to store the extracted vertical lines
        var verticalLines = new List<Mat>();

        // Process each frame
        Mat frame = new Mat();
        while (capture.Read(frame))
        {
            if (frame.Empty())
                break;

            // Get the middle column of the frame
            int middleColumn = frame.Cols / 2;
            var middleColumnMat = frame.Col(middleColumn); // Extract middle column

            // Add the column to the list
            verticalLines.Add(middleColumnMat.Clone()); // Clone to ensure memory safety
        }

        // Determine height of the stitched image
        int stitchedHeight = verticalLines[0].Rows; // All columns have the same height
        int stitchedWidth = verticalLines.Count;    // Number of columns

        // Create an empty Mat for the stitched image
        var stitchedImage = new Mat(stitchedHeight, stitchedWidth, MatType.CV_8UC3);

        // Populate the stitched image with vertical lines
        for (int i = 0; i < verticalLines.Count; i++)
        {
            var column = verticalLines[i];
            column.CopyTo(stitchedImage.Col(i)); // Copy column to the corresponding position
        }

        // Save the stitched image
        Cv2.ImWrite("stitched_image.png", stitchedImage);

        Console.WriteLine("Stitched image saved as 'stitched_image.png'.");
    }
}

“DllNotFoundException: Unable to load DLL ‘OpenCvSharpExtern’ or one of its dependencies: The specified module could not be found. “

So it directed me to install the required runtime.

“Can I put a marker at the bottom of the image every 5 seconds of video, assuming 30fps means every 150 lines”

This then built up to ticks every 1,5 and 10 seconds and every minute. The story goes on ..

A WPF app was requested that called upon the stiching class in the Console app displaying teh image. The sticthing class was then refactored into a separate project and the mouse cursor based timing process was created with a number of queries.

A MAUI app was then added and runng on a Windows desktop was able to do the image stitchinhg and display the result. But when the target was an Android phone, the lack of an Android runtime in shimat/opencvsharp repository caused teh MAUI app to not be able to run on th phone. There was a suggested third party runtime, but could get that to work.

At times I had to tell Copilot that what it proposed didn’t work to which it suggested alternatives. Other times I had to work it out myself with a few tries at possible coding (Intellisense property and method suggestions of entities) or search elsewhere. A couple of times I rewind a little and went in slightly different direction.

Great work identifying and implementing that tweak!

Engu.CV

Some searching and a query to other MVPs suggested ENGU.CV. There is a managed library.

I was able to ask Copilot to to port the Stitching class to use ENGU.CV. It seem to know about and was able, in teh main to port it. There were some sticking points where it used OpenCv properties and methods that did not exist with Engu.CV. It would then sugest chenges but ended goibg around in circles. Thus I had reached teh limit of GitHub Copilot. I guess it uses publically available code and so if it hadn’t been done in that context then there wasn’t a solution available.

I then consulted the Engu.CV documentation and with a bit of drilling into that and a few tries I was able to get the image stitching runing on the phone in the mAUI app. :)

    var column = verticalLines[i];
    column.CopyTo(stitchedImage.RowRange(0, stitchedHeight).Col(i));

in OpneCVSharp became:

using (Mat middleLine = frame.Col(middleColumn))
{

    // Append the middle line to the stitched image
    if (stitchedImage.IsEmpty)
    {
        stitchedImage = middleLineMod.Clone();
    }
    else
    {
        CvInvoke.HConcat(new Mat[] { stitchedImage, middleLineMod }, stitchedImage);
    }
}
CvInvoke.Imwrite(outputFilepath, stitchedImage);

in Engu.CV

The time ticks at the bottom of the image were implemented quite differently with the 2 solutions:

// Draw tick as a vertical line, color and length depending upon time tick type. 1 second tick shown

 var stitchedImage = new Mat(stitchedHeight + 100,  ... //Add ticks area at bottom
 ...
 if (i % Fps == 0) // Black 1-second marker 12px
{
    Cv2.Line(stitchedImage, new Point(i, stitchedHeight), new Point(i, stitchedHeight + 12), new Scalar(0, 0, 0), 1); // black line
}

in OpneCVSharp became:

// Overwrite ticks area at bottom
// Have to write pixel by pixel
int ht = 100; // Height at bottom of ticks area at bottom.Ticks go down.
int tick = 0; // Height of tick, different for whether 1,5,10 sec or 1 minute
int startPix = img.Height - ht;
....
if (i % Fps == 0) // Black 1-second marker
{
    tick = ht / 3;
    bgr = new Bgr(0, 0, 0); // Black color
}
...
tick = startPix + tick;
...
for (int ii = startPix; ii < img.Height; ii++)
{
    if (ii < tick) 
        img[ii, 0] = bgr;
    else
        img[ii, 0] = white; // Set to white
}

in Engu.CV

Outcome

In less than a week I have a working app in 3 contexts. There have been 90+ downloads by 60+ participants!

All good

 TopicSubtopic
<  Prev:   Azure Devops
   
 This Category Links 
Category:Coding Index:Coding
<  Prev:   AI