Setup e2e testing for live preview features

[ang-zeyu]

Is your request related to a problem?

With #1513, #1529, etc., there's quite a chunk of logic concerning live reload now.

Describe the solution you'd like

Would be good to explore and add in some e2e live-preview testing to prevent regressions.

[tlylt]

Will Cypress be a good option to explore, in addition to Puppeteer? I have previously used Cypress for frontend testing and it is getting very popular in recent times.

I am a bit confused as to how to test a live-reload functionality. Do you have any rough idea as to how it can be implemented?
It seems to me that we need to perhaps:

• start running the server and serve the page on a browser
• start running a test case on a page
• somehow modify a file and save a file, and that triggers the reloading of the page
• cypress(or others) while the current test case is ongoing, picks up the changes in the page and verify that something has been updated

I will try to research more on this in the meantime :)

[ang-zeyu]

My primary concern with cypress would just be cost; There's an open source plan but it I'd rather go for something free from the offset.
Puppeteer is also maintained by the chrome devtools team so we should be in good hands here =P

There's also the lesser issue of cypress startup being much longer. I haven't used cypress in a while though so I'm not sure if that's changed.

Feel free to explore cypress still though, there might be some other benefits I'm not aware of. (e.g. if puppeteer dosen't support the necessary apis)

I am a bit confused as to how to test a live-reload functionality. Do you have any rough idea as to how it can be implemented?
It seems to me that we need to perhaps:

Pretty much what I had in mind for now!

A little clunky but since we are testing filesystem event handlers / websockets / etc. it might be best to replicate user action as much as possible. (e.g. fs.writeFileSync to trigger the change)

[tlylt]

My primary concern with cypress would just be cost; There's an open source plan but it I'd rather go for something free from the offset. Puppeteer is also maintained by the chrome devtools team so we should be in good hands here =P

I see. If I'm not wrong, we can use Cypress with its test runner and do away with the dashboard service as that is the paid part and quite secondary. Some ref here and here. I think the test case stats can still be collected in CI, while the dashboard service just makes the tracking and viewing of such data more pleasant.

There's also the lesser issue of cypress startup being much longer. I haven't used cypress in a while though so I'm not sure if that's changed.

Oh, interesting comparison! I didn't know about this before but I do feel that running test cases with Cypress can take a while, which I assumed to be due to the inherent "lagginess" of frontend testing 😂

[tlylt]

This issue is now up for grabs 🤖.

I did some work in #1763 a while back, trying to set up our e2e test suite using jest-puppeteer. However, there were a few issues (likely still around) that made the change difficult. I have decided to thus close the WIP PR, mainly because:

• jest puppeteer does not have built-in support for an easy way to test multiple combination of cli commands to spin up a server, which may make things difficult if we want to comprehensively test things like markbind serve -o, markbind serve -d, markbind serve -d -o. While we can try to work around it by creating a script to do so, my experiments so far were unsuccessful.
• even the simplest e2e tests may fail unexpectedly. The flakiness is quite hard to detect in that at times the tests would run successfully in multiple tries locally, and yet it fails on CI. Some of the issues were discussed in the PR comments, but it seems to be due to some race conditions when files are modified at the same time.