Rework che-theia build process to not use source linking

[tsmaeder]

Currently, we're linking source folders in order build che theia, we have the following structure ('->' being a symlink):

/theia
  /.git
  ...
  /plugins
    ->/theia/che/che-theia/plugins/task-plugin
    ...
  /extensions
    ->/theia/che/che-theia/extensions/eclipse-che-theia-about
  /che
    /che-theia
    /.git
    ...
    /plugins
      /task-plugin
    /extensions
      /eclipse-che-theia-about

The problem with this is that the build process does not handle source links very well: for one things, there will be two node_modules folder: one inside /theia and one inside /theia/che/che-theia. The fact that cd /projects/theia && yarn becomes broken when you rebuild a che plugin or extension seems to be a consequence of this (deleting both node_modules directories fixes the problem).
Since the build only works if the che-theia folder is located inside the theia folder, there is an additional drawback: the theia git tools cannot be used for changes in che-theia, because theia can't handle the case of nested git repos, it seems.

My proposal is to move to a setup that is based on yarn link. The idea is to have che-theia as a separate repository and to yarn link $package the theia packages in the appropriate places. The setup would look like this:

/theia
...just a vanilla theia install with all the packages linked to ~/.config/yarn/link
/che-theia
/assembly
...
/extensions
...
/plugins
...

I was able to almost make this work in a day an it seemed to address the problems mentioned above. I think investing some more time would be worth the time.

[benoitf]

I think it's something that can be done properly/smoothly only with yarn 2

[tsmaeder]

Let's give it a try, shall we?

[tsmaeder]

And why would it not work with yarn 1.x?

[tsmaeder]

@benoitf ?

[benoitf]

One issue is yarn link is considering links to be folders, not packages
so there are issues with transitive dependencies / peer dependencies
yarnpkg/yarn#2914

portal: might be the savior:
https://dev.to/arcanis/introducing-yarn-2-4eh1#new-protocol-raw-portal-endraw-

https://yarnpkg.com/features/protocols#whats-the-difference-between-link-and-portal

Also the fact that we can store the dependencies should improve velocity.

[tsmaeder]

@benoitf I've read the linked issue, but I don't really understand why it would block us: the issues seems to state the yarn does not call install on the library packages (in our case: the theia packages). But we could do that ourselves when necessary.
What we need is this:

1. For our packages to be compiled against the correct version of upstream packages, assuming we have built those correct.y
2. When packaging a build target with webpack or other tools, we need to pick up all dependencies and the proper version of the dependencies

Could you spell out how the yarn link behavior is breaking those requirements? It is not clear to me from reading the issue.