Compilation errors don't cause non-zero exit codes in ng test

[JohannesRudolph]

Please provide us with the following information:

OS?

Windows 7, 8 or 10. Linux (which distribution). Mac OSX (Yosemite? El Capitan?)
OS X El Capitan

Versions.

Please run ng --version. If there's nothing outputted, please run in a Terminal: node --version and paste the result here:

angular-cli: 1.0.0-beta.17
node: 6.3.0
os: darwin x64

Repro steps.

Was this an app that wasn't created using the CLI? What change did you do on your code? etc.
Create a angular cli project, create a compilation error in a spec file that doesn't fail at runtime (e.g. omit constructor parameter) and run ng test

The log given by the failure.

Normally this include a stack trace and some more information.

For example, this output of a test run of ng test:

ERROR in [default] /Users/jr/dev/console/console/src/app/context/context.service.spec.ts:17:12 
Supplied parameters do not match any signature of call target.

ERROR in [default] /Users/jr/dev/console/console/src/app/context/location.spec.ts:8:11 
Supplied parameters do not match any signature of call target.
19 10 2016 14:05:24.377:INFO [karma]: Karma v1.2.0 server started at http://localhost:9876/
19 10 2016 14:05:24.380:INFO [launcher]: Launching browser Chrome with unlimited concurrency
19 10 2016 14:05:24.511:INFO [launcher]: Starting browser Chrome
19 10 2016 14:05:25.882:INFO [Chrome 53.0.2785 (Mac OS X 10.11.6)]: Connected on socket /#pLo6aNiQB7OJVlxvAAAA with id 31332214
Chrome 53.0.2785 (Mac OS X 10.11.6): Executed 10 of 10 SUCCESS (0.77 secs / 0.754 secs)
iDevBook01:console jr$ $?
-bash: 0: command not found

Mention any other details that might be useful.

It would be useful if angular-cli captured the error code when compiling typescript code and failed early. ng build doesn't build tests, so it's not useful either.

---

Thanks! We'll be in touch soon.

[kylecordes]

This also affects, for example, ng build. An argument could be made that ng serve should carry on even with typescript errors, but build and test presumably should fail if the code is broken.

[filipesilva]

Commands like ng test, ng serve and ng build --watch that run in watch mode don't usually exit with an error. This is by design.

Non-watch commands like ng test --watch=false, ng build and ng e2e do exit with an error code though, so I think what you're looking for is ng test --watch=false.

For instance, I removed AppComponent parameters in let fixture = TestBed.createComponent(AppComponent);, which causes the Supplied parameters do not match any signature of call target. error:

kamik@T460p MINGW64 /D/sandbox/master-proj (master)
$ ng test --watch=false


ERROR in [default] D:\sandbox\master-proj\src\app\app.component.spec.ts:16:18
Supplied parameters do not match any signature of call target.
27 10 2016 16:22:03.288:INFO [karma]: Karma v1.2.0 server started at http://localhost:9876/
27 10 2016 16:22:03.290:INFO [launcher]: Launching browser Chrome with unlimited concurrency
27 10 2016 16:22:03.297:INFO [launcher]: Starting browser Chrome
27 10 2016 16:22:04.628:INFO [Chrome 54.0.2840 (Windows 10 0.0.0)]: Connected on socket /#cdGZJk8pod052mpYAAAA with id 64527908
Chrome 54.0.2840 (Windows 10 0.0.0) App: MasterProj should create the app FAILED
        Failed: Cannot create the component undefined as it was not imported into the testing module!
        Error: Cannot create the component undefined as it was not imported into the testing module!
            at TestBed.createComponent (webpack:///D:/sandbox/master-proj/~/@angular/core/bundles/core-testing.umd.js:846:0 <- src/test.ts:3863:23)
            at Function.TestBed.createComponent (webpack:///D:/sandbox/master-proj/~/@angular/core/bundles/core-testing.umd.js:673:0 <- src/test.ts:3690:33)
            at webpack:///D:/sandbox/master-proj/src/app/app.component.spec.ts:16:26 <- src/test.ts:21280:41
            at ZoneDelegate.invoke (webpack:///D:/sandbox/master-proj/~/zone.js/dist/zone.js:232:0 <- src/test.ts:20064:26)
            at AsyncTestZoneSpec.onInvoke (webpack:///D:/sandbox/master-proj/~/zone.js/dist/async-test.js:49:0 <- src/test.ts:12928:39)
            at ProxyZoneSpec.onInvoke (webpack:///D:/sandbox/master-proj/~/zone.js/dist/proxy.js:76:0 <- src/test.ts:13620:39)
            at ZoneDelegate.invoke (webpack:///D:/sandbox/master-proj/~/zone.js/dist/zone.js:231:0 <- src/test.ts:20063:32)
            at Zone.runGuarded (webpack:///D:/sandbox/master-proj/~/zone.js/dist/zone.js:128:0 <- src/test.ts:19960:47)
            at runInTestZone (webpack:///D:/sandbox/master-proj/~/@angular/core/bundles/core-testing.umd.js:105:0 <- src/test.ts:3122:29)
            at Object.<anonymous> (webpack:///D:/sandbox/master-proj/~/@angular/core/bundles/core-testing.umd.js:48:0 <- src/test.ts:3065:17)
Chrome 54.0.2840 (Windows 10 0.0.0): Executed 1 of 3 (1 FAILED) (0 secs / 0.093 secs)
Chrome 54.0.2840 (Windows 10 0.0.0) App: MasterProj should create the app FAILED
        Failed: Cannot create the component undefined as it was not imported into the testing module!
        Error: Cannot create the component undefined as it was not imported into the testing module!
            at TestBed.createComponent (webpack:///D:/sandbox/master-proj/~/@angular/core/bundles/core-testing.umd.js:846:0 <- src/test.ts:3863:23)
            at Function.TestBed.createComponent (webpack:///D:/sandbox/master-proj/~/@angular/core/bundles/core-testing.umd.js:673:0 <- src/test.ts:3690:33)
            at webpack:///D:/sandbox/master-proj/src/app/app.component.spec.ts:16:26 <- src/test.ts:21280:41
            at ZoneDelegate.invoke (webpack:///D:/sandbox/master-proj/~/zone.js/dist/zone.js:232:0 <- src/test.ts:20064:26)
            at AsyncTestZoneSpec.onInvoke (webpack:///D:/sandbox/master-proj/~/zone.js/dist/async-test.js:49:0 <- src/test.ts:12928:39)
            at ProxyZoneSpec.onInvoke (webpack:///D:/sandbox/master-proj/~/zone.js/dist/proxy.js:76:0 <- src/test.ts:13620:39)
            at ZoneDelegate.invoke (webpack:///D:/sandbox/master-proj/~/zone.js/dist/zone.js:231:0 <- src/test.ts:20063:32)
            at Zone.runGuarded (webpack:///D:/sandbox/master-proj/~/zone.js/dist/zone.js:128:0 <- src/test.ts:19960:47)
            at runInTestZone (webpack:///D:/sandbox/master-proj/~/@angular/core/bundles/core-testing.umd.js:105:0 <- src/test.ts:3122:29)
Chrome 54.0.2840 (Windows 10 0.0.0): Executed 3 of 3 (1 FAILED) (0.211 secs / 0.18 secs)

kamik@T460p MINGW64 /D/sandbox/master-proj (master)
$ echo $?
1

@kylecordes I don't understand what you're talking about. Have you tried ng build with a compilation error and did not get an error exit code? If so, please open a new issue with a reproduction.

[JohannesRudolph]

Thanks for responding and your great work on angular-cli @filipesilva

sorry for not being more specific in the original request. The error is a build error not a test error. I did indeed execute the test with ng test --watch=false and observed that the exit code is zero (see below for a full reproduction).

I agree about returning nothing meaningful in --watch=true, but the original issue still stands. Is that enough for reopening this issue?

iDevBook01:console jr$ ng test --watch=false
Could not start watchman; falling back to NodeWatcher for file system events.
Visit http://ember-cli.com/user-guide/#watchman for more info.

WARNING in ./src/app/core/location.ts
[5, 85]: missing whitespace

 @ ./src/app/core/index.ts 9:451-472
 @ ./src/app/app.component.ts
 @ ./src/app/app.component.spec.ts
 @ ./src \.spec\.ts
 @ ./src/test.ts

ERROR in [default] /Users/jr/dev/console/console/src/app/core/context.service.spec.ts:24:12 
Supplied parameters do not match any signature of call target.

28 10 2016 15:36:40.593:INFO [karma]: Karma v1.2.0 server started at http://localhost:9876/
28 10 2016 15:36:40.595:INFO [launcher]: Launching browser Chrome with unlimited concurrency
28 10 2016 15:36:40.601:INFO [launcher]: Starting browser Chrome
28 10 2016 15:36:42.874:INFO [Chrome 54.0.2840 (Mac OS X 10.11.6)]: Connected on socket /#qitPU5ll3ETn1TolAAAA with id 88987585
Chrome 54.0.2840 (Mac OS X 10.11.6): Executed 3 of 18 SUCCESS (0 secs / 0.516 secs)
....
iDevBook01:console jr$ $?
-bash: 0: command not found

[filipesilva]

A build error should definitely make ng test --watch=false exit with an error code. In my test it did, but in your case it did not even though they should have been very similar.

Can you give me the exact code I can use to reproduce?

[fabian-dev]

I can confirm this.

In a new angular-cli Project create a spec that has TS errors, but is valid JS. Example app.component.spec.ts:

describe('App: Ngnew', () => {
  beforeEach(() => {
    TestBed.configureTestingModule({
      declarations: [
        AppComponent
      ],
    });
  });

  export interface UseInSut {
    someValue: number;
  }

  it('should create the app', async(() => {
    let useInSut: UseInSut = {
      nonExisitung: 1;
    }
    expect(true).toBeTruthy();
  }));

});

Although not valid TS, ng test will pass with RC=0.

$ ng test --watch=false

ERROR in [default] C:\DEV\TRY\ngnew\src\app\app.component.spec.ts:23:6
Type '{ nonExisitung: number; }' is not assignable to type 'UseInSut'.
  Object literal may only specify known properties, and 'nonExisitung' does not exist in type 'UseInSut'.

ERROR in [default] C:\DEV\TRY\ngnew\src\app\app.component.spec.ts:23:21
',' expected.
18 11 2016 14:10:09.677:INFO [karma]: Karma v1.2.0 server started at http://localhost:9876/
18 11 2016 14:10:09.679:INFO [launcher]: Launching browser Chrome with unlimited concurrency
18 11 2016 14:10:09.687:INFO [launcher]: Starting browser Chrome
18 11 2016 14:10:12.066:INFO [Chrome 54.0.2840 (Windows 7 0.0.0)]: Connected on socket /#YuC0qEmPAM5CAwZoAAAA with id 63478678
Chrome 54.0.2840 (Windows 7 0.0.0): Executed 1 of 1 SUCCESS (0.009 secs / 0.005 secs)

afterwards $ echo $? gives me 0.

I would expect ng test to fail on TS errors.