Gatsby's Use Of Polyfills and the 'Missing Resources for x' error

[Undistraction]

Description

I've just been through the incredibly painful process of getting a Gatsby site working in IE 11 and thought it was worth writing up the problems I faced due to Gatsby's current architecture.

We were receiving an error for every page in IE:

Can't find resource for ...

There was no other error. The pages loaded fine in all the other browsers we tested, so my hunch was that Gatsby was swallowing some kind of IE-specific error.

After a lot of digging I eventually tracked the point of failure to Gatsby's loader, specifically, this line.

.catch(() => {
   failed = true
})

Inspecting the value passed to the catch I could see that the error was related to something that needed to be polyfilled. The problem here being that Gatsby was swallowing this useful error and giving me nothing to help me pinpoint the issue. The resource wasn't missing - it was just erroring when the Loader tried to fetch it. It would make a lot of sense to error here with a more useful message, passing on the specific error that the loader had encountered.

This leads me on to the second issue. Gatsby uses babel-preset-env to handle polyfills, and by default useBuiltIns is set to usage. This sounds great on the surface, but masks what I would consider to be a big problem. If any third-party modules need polyfilling, it seems that Gatsby will not add the Polyfills they need. It will add polyfill imports to project-based code, but will not add imports to non-project-based code. This means if a third-part module was published with useBuiltIns set to entry, it appears that Gatsby will not include the expected babel-polyfill import and replace with the necessary polyfill imports. This means that if a third-party library needs a polyfill that hasn't been supplied due usage within the gatsby project, then this polyfill will not be included. In my case it was a dependency of another dependency.

Combined with the first issue, this was nightmarishly hard to debug.

I'd also suggest adding a section to the browser support docs that makes it clear that Gatsby's declared browser support is only for production builds. I spent a good deal of time trying to debug IE locally before realising that Gatsby doesn't support IE11 when running in development. I eventually found the docs for Gatsby's default babel preset which explained this, but I think these need to be much more prominent. To anyone encountering a similar issue, the only fix I've found is to manually include the entire babel-polyfill using:

// gatsby-browser.js

require('babel-polyfill')

exports.onClientEntry = () => {
  // Without this function body the import will not be picked up.
}

This is far from ideal as this includes a tonne of extra polyfills we don't need, but it at least fixes the problem in the short term.

I guess another option would be to switch Gatsby from using usage to entry so that the import will be replaced by only required polyfills. That appears to be possible by overriding Gatsby's default babel config.

The above is true as of "gatsby": "2.1.22"

[KyleAMathews]

Hey, sorry you had so much trouble :-( this is definitely something we should fix.

The problem I believe isn't Babel but the fact that we don't transpile node_modules. Which means that any code there which should be polyfilling, Babel doesn't see so doesn't add the polyfills.

[DSchau]

First of all - thanks for writing this up! We appreciate it, and as @KyleAMathews said, we're bummed to hear about the pain points you've been experiencing.

Let's dive into some of your points in a bit more detail, though!

Inspecting the value passed to the catch I could see that the error was related to something that needed to be polyfilled. The problem here being that Gatsby was swallowing this useful error and giving me nothing to help me pinpoint the issue. The resource wasn't missing - it was just erroring when the Loader tried to fetch it. It would make a lot of sense to error here with a more useful message, passing on the specific error that the loader had encountered.

Would you be interested in PRing a change for logging the error? I think this is a pretty non-intrusive change that will be impactful for a lot of users testing in e.g. IE 11.

If any third-party modules need polyfilling, it seems that Gatsby will not add the Polyfills they need. It will add polyfill imports to project-based code, but will not add imports to non-project-based code

This is correct, yeah. One thing we've considered is transpiling node_modules, but there are trade-offs there as well, primarily related to performance and possible redundancy. I believe create-react-app does this starting in v2, so it's certainly something we should investigate.

As @KyleAMathews notes though, babel-preset-env is only able to operate on userland code, e.g. your src/ directory. We'd need to start transpiling node_modules for the usage setting to be most valuable.

Finally:

I spent a good deal of time trying to debug IE locally before realising that Gatsby doesn't support IE11 when running in development

Would you happen to have any more info on what failed? I'd imagine we can include some polyfills in development only so that we can avoid this. We actually already do something similar here

In effect - we can absolutely harden our support for IE 11. We have some low-hanging fruit that can be PR'd to improve upon the development experience and alert of issues early. Longer term - transpiling node_modules may be a good solution for more of these issues.

Thanks for opening this!

[Undistraction]

Just one of those things. The fact Gatsby was building without error, then swallowing the runtime error was what made it really horrible to debug, so I think that would be a useful fix.

In the short term what would you think the best approach is to involve including the whole of babel-polyfill just to cover node modules that need to be polyfilled? Do you think that overriding Gatsby's default Babel config and using entry instead is the best approach?

[DSchau]

In the short term what would you think the best approach is to involve including the whole of babel-polyfill just to cover node modules that need to be polyfilled?

Do you mean for your particular use case, or globally? I can chime in on #12409 with some more thoughts.

[Undistraction]

@DSchau Sorry. Think we overlapped there.

Would you be interested in PRing a change for logging the error? I think this is a pretty non-intrusive > change that will be impactful for a lot of users testing in e.g. IE 11.

Yes of course. I'll try and make some time this week.

Would you happen to have any more info on what failed? I'd imagine we can include some polyfills in > development only so that we can avoid this. We actually already do something similar here

My understanding from the docs for babel-preset-gatsby is that Gatsby conciously doesn't support IE in development. It says of its default targets:

defaults to { "browsers": ["last 4 versions", "safari >= 7", "ie >= 9"] } in production and { "browsers": ["last 2 versions", "not ie <= 11", "not android 4.4.3"] } in development when targeting the browser.

So in terms of polyfilling userland code, I guess the solution is to support the same in development as in production, or at least provide a simple way to add this support.

Do you mean for your particular use case, or globally? I can chime in on #12409 with some more thoughts.

Would be great to hear your thoughts for the best way to handle this in my case. I definitely have multiple node_modules that need to be polyfilled. Not sure what the best approach there is. Is it to include babel-polyfill and override Gatsby's default babel config to make sure it uses entries so that it includes all polyfills for the specified range?

[DSchau]

I definitely have multiple node_modules that need to be polyfilled

The quickest here is to just conditionally enable transpilation on those modules, with a plugin like gatsby-plugin-compile-es6-modules.

You'd use it like so in gatsby-config.js:

module.exports = {
  plugins: [
    {
      resolve: `gatsby-plugin-compile-es6-packages`,
      options: {
        modules: [
          'module-1',
          'module-2',
          'module-you-get-the-idea'
        ]
      }
    }
  ]
}

[Undistraction]

@DSchau Thanks. That looks like a good option. Guess the problem there is how easy it would be for a new module that needs polyfilling to slip in as a dependency or sub-dependency without being added to the list.

[DSchau]

@Undistraction I suppose that will just require a little due diligence in ensuring that doesn't happen! You could probably automate the check with a CI task and some type of traversal of code you depend upon.

Is this happening fairly frequently? More specifically--which packages does this seem to be occurring in?

[Undistraction]

@DSchau Yep. That's a fair point. There were a few places I found it was happening. I was using Formik which in turn uses Yup. Yup uses methods on Set that need to be polyfilled for IE11. I was also using our own packages from NPM (from a monorepo). They are all transpiled with babel (with modules left as ES6 so Gatsby can use Tree Shaking). They use a few features that need to be polyfilled.

[Undistraction]

This is what I ended up doing:

I opted not to use gatsby-plugin-compile-es6-packages as I don't think this approach scales. It might if the only modules that were causing problems were top level dependencies, but given that it could be a module deep in the dependency tree that needs polyfilling, I feel this is an ineffective solution. I was also concerned with potential side-effects by transpiling third-party libraries.

I chose to go with using 'entry' instead because I can be sure that they correct polyfills will be present for my chosen browser range, regardless of whether those polyfills are needed within my gatsby project. The other option is to include the whole polyfill, but that will add ~100kb to your project.

1. Create a .babelrc at the root of your gatsby project. This is based on gatsby's babel-preset-gatsby, however I couldn't get gatsby to recognise either babel.config.js or babelrc.js so I had to use a JSON format. Note that I've set debug to true for babel-preset-env which will output a lot of useful information including a list of which transforms are being applied and which polyfills have been used to replace your import '@babel/polyfill' :

{
  "env": {
    "test": {
      "presets": [
        "@babel/preset-env",
        {
          "loose": true,
          "modules": "commonjs",
          "useBuiltIns": "entry",
          "targets": { "browsers": [">0.25%", "not dead"] }
        }
      ]
    }
  },
  "presets": [
    "babel-preset-gatsby",
    [
      "@babel/preset-env",
      {
        "debug": true,
        "loose": true,
        "modules": false,
        "useBuiltIns": "entry",
        "targets": { "browsers": [">0.25%", "not dead"] }
      }
    ]
  ],
  "plugins": [
    [
      "@babel/plugin-proposal-class-properties",
      {
        "loose": true
      }
    ],
    "babel-plugin-macros",
    "@babel/plugin-syntax-dynamic-import",
    [
      "@babel/plugin-transform-runtime",
      {
        "helpers": true,
        "regenerator": true
      }
    ]
  ]
}

2. Add the following to gatsby-browser.js in the root of your gatsby project. Note that you must use ES6 imports otherwise the polyfill will not be picked up.

import '@babel/polyfill'

// eslint-disable-next-line import/prefer-default-export
export const onClientEntry = () => {
  // Without this function body the import will not be picked up.
}

That should get things working. I'm not going to close this as I believe Gatsby's current approach is not the right one. It makes it very easy to end up with a broken site in some browsers just to make a relatively small size gain through using 'usage' instead of 'entry'. Personally I see this as a poor tradeoff, and most importantly an unexpected issue and one that is hard to debug.

[DSchau]

This is based on gatsby's babel-preset-gatsby, however I couldn't get gatsby to recognise either babel.config.js or babelrc.js

Did you try .babelrc.js? If that doesn't work--that's a bit concerning. If .babelrc works, though, then the .js variant should as well.

I was also concerned with potential side-effects by transpiling third-party libraries

You see our concern with us transpiling node_modules globally, then! Generally - I think this is pretty safe, but I see your point.

I'm not going to close this as I believe Gatsby's current approach is not the right one

Kinda agree with this. I think smart defaults but also allowing these defaults to be customized is the right approach. I like your idea in #12409 in re: to this.

Do you think a reasonable solution is to implement a PR that satisfies #12409 and improved documentation, here?

[Undistraction]

You see our concern with us transpiring node_modules globally, then! Generally - I think this is pretty safe, but I see your point.

I completely see your concern. I'm absolutely not being down on you guys. It's a tricky problem to solve, and getting into transpiling third party modules is a big thing and one that will bring its own batch of obscure issues.

Do you think a reasonable solution is to implement a PR that satisfies #12409 and improved documentation, here?

I think that would be helpful yes.