Project and sources to build Runtests.exe ?

[doingnz]

Where can I find how to build and debug RunTests.exe?

I can run individual tests using the supplied RunTests.exe OK, but occasionally get errors running all tests with timeouts between test solutions running on the live hardware. I hope to make a modified RunTests.exe that can cope with longer delays between tests. Looking for the ideal SIG (All green) on the unit tests Results.htm.

Thanks
Richard

[mortezag]

The project file corresponding to RunTests.exe is at Test\Platform\Tools\MFTestSystem\MFTestSystem.csproj. (Generally, the relevant source code files are at Test\Platform\Tools)

By the way, you should notice that not all the unit tests are passing at the moment but a big majority are. In fact, there are a few test bugs which we are investigating at the moment.

[doingnz]

Thank you. Not sure how I missed that (Blush!)

I have been working on the presentation.sln Unit Tests.

I suggest that the events are added to the Child of the mainWindow, not directly to mainWindow object as there is no method to remove event handlers once added that I am aware of. The CleanWindow() method allocates a new Text child, effectively removing the event handlers.
i.e.

    object RaiseEvent(object obj)
    {
        bool bRet = false;
        mainWindow.Child.RaiseEvent((RoutedEventArgs)obj);
        bRet &= _handlerEvent.WaitOne(1000, true);
        return null;
    }

and

    object AddToHandler(object obj)
    {
        if (obj is RoutedEvent && btnEvent)
        {
            mainWindow.Child.AddHandler((RoutedEvent)obj, new RoutedEventHandler(HandleButtonEvents), handledEventsToo);
        }
        if (obj is RoutedEvent &&  fcsEvent)
        {
            mainWindow.Child.AddHandler((RoutedEvent)obj, new RoutedEventHandler(HandleFocusChangedEvent), handledEventsToo);
        }
        if (obj is EventRoute)
        {
            RoutedEventArgs args = null;

            if (btnEvent)
            {
                args = new ButtonEventArgs(null, null, DateTime.Now, Hardware.Button.AppDefined1);
                args.RoutedEvent = Buttons.ButtonDownEvent;
            }
            else
            {
                args = new FocusChangedEventArgs(null, DateTime.Now, mainWindow, mainWindow);
                args.RoutedEvent = Buttons.LostFocusEvent;
            }
            mainWindow.Child.AddToEventRoute((EventRoute)obj, args);
        }
        return null;
    }

Then add the following snippet to the following tests at the end of each test to clean up and remove the event handlers.

        if (CleaningWindow() != MFTestResults.Pass)
        {
            return MFTestResults.Fail;
        }

public MFTestResults UIElement_RaiseEvent_ButtonEvent_Test1()
public MFTestResults UIElement_RaiseEvent_FocusChangedEvent_Test2()
public MFTestResults UIElement_AddHandlerTest3()
public MFTestResults UIElement_AddToEventRouteTest4()

I have not been able to fix public MFTestResults UIElement_ArrangeTest4(). Previously it was working due to the event handlers still being present from previous tests.

Specifically the following code does not appear to achieve the desired result.

            eRoute = new EventRoute(rEvents[i]);
            mainWindow.Dispatcher.Invoke(new TimeSpan(0, 0, 10),
                    new DispatcherOperationCallback(AddToHandler), eRoute);

This test 4 will only pass if they are replaced with

            mainWindow.Dispatcher.Invoke(new TimeSpan(0, 0, 5), new DispatcherOperationCallback(AddToHandler), rEvents[i]);

Same for

            eRoute = new EventRoute(rEvents[i]);
            mainWindow.Dispatcher.Invoke(new TimeSpan(0, 0, 10),
                new DispatcherOperationCallback(AddToHandler), eRoute);

to be replaced with

           mainWindow.Dispatcher.Invoke(new TimeSpan(0, 0, 5), new DispatcherOperationCallback(AddToHandler), rEvents[i]);

I am not up to speed with EventRoute, so not sure of the fix.

In test public MFTestResults UIElement_InvalidateTest16(), I had to use mainWindow Dispatcher to get it to update the rectangle? when using _rect object, the method InvalidateTest was never called.

// _rect.Dispatcher.Invoke(new TimeSpan(0, 0, 20),
// new DispatcherOperationCallback(InvalidateTest), null);
mainWindow.Dispatcher.Invoke(new TimeSpan(0, 0, 20),
new DispatcherOperationCallback(InvalidateTest), null);

In public MFTestResults UIElement_InvalidateArrangeTest17(), you can change
MFTestResults testResult = MFTestResults.Skip; to Pass.if you change _rect to mainWindow.Dispatcher

I changed the following two methods to force the rectangle to be updated.

    object InvalidateTest(object obj)
    {
        _rect.Fill = new SolidColorBrush(_color);
        _rect.InvalidateArrange();
        _rect.UpdateLayout();
        _rect.Invalidate();
        return null;
    }

    object InvalidateArrangeTest(object obj)
    {
        _rect.VerticalAlignment = VerticalAlignment.Top;
        _rect.HorizontalAlignment = HorizontalAlignment.Right;
        _rect.InvalidateArrange();
        _rect.UpdateLayout();
        return null;
    }

Let me know if there are any questions.

Kind regards
Richard
PS: testing above in MF 4.1 as production still MF 4.1 Starting to play with MF 4.4 in a virtual machine....

[doingnz]

It would be good for the Wiki ("Getting Started"?) to be extended to describe and recommend the processes to run the Tests.

Adds some focus on their on-going value. Maintaining and developing them ensures we don't inadvertently break anything along the way.

I assume the correct way to build the tests is.

msbuild build.dirproj

To Run the tests

cd %BUILD_TEST_TREE_SERVER%\dll
RunTests

[doingnz]

What is the correct way to load the Solutions for individual Tests into VS2015 so they can be run and debugged? A related question is the purpose of setenv_vs? When should this be used?

The .csproj files need the following to be defined for the HintPath.

BUILD_TREE_DLL
BUILD_TEST_TREE_CLIENT

e.g..

      <HintPath>$(BUILD_TREE_DLL)\Microsoft.SPOT.Native.dll</HintPath>
and
      <HintPath>$(BUILD_TEST_TREE_CLIENT)\Microsoft.SPOT.Platform.Test.MFTestRunner.dll</HintPath>

BUILD_TEST_TREE_CLIENT is defined. And BUILD_TREE_DLL can be created as follows.

set BUILD_TREE_DLL=%BUILD_TREE_CLIENT%\dll

Maybe BUILD_TREE_DLL should be added to setenv_vs or setenv_base?

On Win7, with VS2013 & RVDS I use the following devenv.cmd in my %SPOCLIENT% folder to launch VS2013 in the context of the Cmd prompt with all the environment variables set.

call setenv_RVDS4.1.cmd 
set BUILD_TREE_DLL = %BUILD_TREE_CLIENT%\dll
"%VSINSTALLDIR%\Common7\IDE\devenv.exe"

I can load a Test .sln file from VS2013 File/Project menu and Start it and single step as one would expect.

This does not work on Win10, VS2015 with DS5.(with setenv_ds5 5.05)

VS2015 does not find the Reference assemblies. I have checked they exist in the expected folders. I must be doing something simple that prevents running the test project from VS2015.

Please will anyone else who can run and single step the tests explain what they are doing to make it work.

Thank you.
Richard

[doingnz]

Should the above comment be moved to it's own issue as its a little different to RunTest.exe issue title?

[smaillet-ms]

to load the project files using BUILD_TREE etc. there are two options:

1. run setenv_vs first and, from the same command prompt, run devenv so that it picks up the env variables
2. Modify and submit a pull request for the project so that it references the BuildEnv.props file (See Framework\Debugger\Debugger.csproj and related VS integration support in Framework\CoreDebug and PR Support to build VS integration directly in VS2015 #260 ) allowing you to just double click on the project instead of requiring setenv_vs + devenv in a command prompt. (We've converted the VS integration we use a lot so we can use the experimental hive debug support but haven't converted all of them )

[doingnz]

Framework\Debugger\Debugger.csproj refers to Framework v4.5. Why?

  <PropertyGroup>
    <VSToolsPath Condition="'$(VSToolsPath)' == ''">$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v$(VisualStudioVersion)</VSToolsPath>
    <TargetFrameworkVersion>v4.5</TargetFrameworkVersion>
    <TargetFrameworkProfile />
  </PropertyGroup>

Are there offline versions of the files in folder *framework\IDE\Targets\v4.5* that are different to the files in online folder v4.4?

[smaillet-ms]

The debugger.csproj project is for the desktop components (specifically the Visual Studio extension for debugging NETMF applications) Thus is sets the desktop framework version it is expected to run with.