Iteration 2 - make the infrastructure more generic

[bajtos]

List of things we need to improve in this example to make it a basis for loopback-workspace 3.0.

• 1. Multiple datasources
• 2. Dynamic configuration of datasources
• 3. Reduce boilerplate, auto-load components
  • a. Extract most of api/app.api.js to loopback core.
  • b. Figure out how to simplify html5/app.html5.js
  • c. Refactor custom Grunt tasks into a module
  • d. There should be a convention for adding new things by adding a new file, without modifying any existing code.
• 4. Automated tests
  • Integration tests for the models and the api server
  • Unit-tests for the html5 client which do not communicate with the backend
  • End-to-end tests - html5 client, api server, models, datasources
• 5. Mix and match components

1. Multiple datasources

The current design of api/app.api.js seems to supports a single db datasource. This is a step back from what we already have in loopback-workspace, where datasources.json can define multiple different datasources.

Here is a list of datasources used by a fully-featured application:

• db for the main database
• push for push notifications
• email for sending e-mails
• rest or soap for getting data from legacy systems
• storage for managing images via loopback-storage-service

On the client side, most models will be attached to the same Remoting datasource. The most notable exception is synchronization, it requires several datasources (local storage, remoting).

(SLA-1023)

2. Dynamic configuration of datasources

The current design makes it easy to change the db configuration depending on environment variables. While this is useful for demo purposes, our current direction regarding deployment & operations favours different approach - see the gist.

Configuration should not be present in a node application archive. The more configuration is bound in, the less choice there will be over deployment environment. Even deploying the same app to staging and then production could become impossible.

We need to keep in mind the use case of loopback-workspace, which should be able to load a list of available datasource without the need to build the project and load it to a Node.js process via require('app.js').

(SLA-1023)

3. Reduce boilerplate, auto-load components

The current version contains too much of boilerplate code and makes it impossible to add new features/models/datasources via loopback-workspace.

Here are few generators that cannot be implemented now:

• add a new model
• add a new datasource
• enable explorer (swagger)
• enable authentication

What is needed:

a. Extract most of api/app.api.js to loopback core. Ideally the existing method app.boot can be modified or extended to support the new layout.

b. Figure out how to simplify html5/app.html5.js. The code to require() all models should be generated by gulpfile, same applies to the code calling require() for all angular controller files.

c. (SLA-938) Refactor most of gulpfile into a module gulp-loopback, so that the application gulpfile contains only calls to gulp-loopback and application-specific options.

d. There should be a convention for adding new things by adding a new file, without modifying any existing code. All components (api, html5, web) should support these conventions, either directly in code (think of app.boot) or via a custom build step. (See also b.)

4. Automated tests

The pull request #1 touched the subject lightly, but no solution was implemented.

I would like to see at least these three categories of tests covered by the example project:

a. (SLA-841) Integration tests for the models and the api server, the tests are permitted to use the database (probably the real one, possibly an in-memory replacement). These tests should run on Node.js.

b. (SLA-1019) Unit-tests for the html5 client which do not communice with the backend. These tests should run in a browser via Karma.

c. (SLA-1020) End-to-end tests - html5 client, api server, models, datasources. There are two flavours: in the first one, developers write unit-test-like code, but the communication with backend is not mocked out. The second flavour tests the application via selenium or a similar framework (e.g. angular's protractor). Both flavours require to start the server before running the tests in the browser.

5. Mix and match components

To keep the loopback-workspace code simple, we should come up with a single structure flexible enough to support different kinds of projects:

• full-stack server + client as presented here: api, models, html5, web.
• a pure API server: api, models
• a website with REST API: api, models, web
• (optional) a website without REST API: models, web

The current design is pretty close, but not there yet.

[ritch]

On the client side, all models should be attached to a single Remoting datasource.

I'm afraid it won't always be this simple. Maybe as a default? Take a look at the replication branch for an example of the client attaching models to various data sources.

[ritch]

We need to keep in mind the use case of loopback-workspace,

Yes.

which should be able to load a list of available datasource without the need to build the project and load it to a Node.js process via require('app.js').

I guess it is a lot trickier to map the metadata back to a file if we are loading it from somewhere else. But we have to take into account models / relations / etc defined only at runtime - or design the workspace / ui in a way that makes it clear these aren't taken into account.

[ritch]

3 Reduce boilerplate, auto-load components

Good ideas here.

4 Automated tests

Agreed. Help here would be appreciated.

[rmg]

The runtime configuration described in the gist that advocates an external configuration server is more about hostnames, ports to listen on, and sensitive credentials. The kind of details that should not necessarily be committed to the application's code base.

[bajtos]

On the client side, all models should be attached to a single Remoting datasource.

I'm afraid it won't always be this simple. Maybe as a default? Take a look at the replication branch for an example of the client attaching models to various data sources.

I have updated the issue description.

Yesterday in our chat, we agreed the best solution is to keep the same list of datasources, but allow datasources to have different configuration on the server and on the client.

[bajtos]

Items 1 & 2 were implemented in #7.

[bajtos]

3.a. Extract most of api/app.api.js to loopback core.

Done in #7, #16.

[bajtos]

3.b. Figure out how to simplify html5/app.html5.js

Done in #37.

[bajtos]

3.c. Refactor most of gulpfile into a module

The build uses Grunt now. There are two non-trivial custom tasks (build-config and run) that are probably worth extracting to a new module.

[bajtos]

3.d. There should be a convention for adding new things by adding a new file, without modifying any existing code.

This is done as of #33 and #37.