docs: update creating source plugins guide & add example repo (#22943)

* add example monorepo, make initial updates for source plugin guide and move some content to the transformer guide

* update sample code in guide for proactive fetching

* update READMEs and remove unused files

* remove more unused files

* Apply suggestions from code review

Co-Authored-By: LB <laurie@gatsbyjs.com>

* split terminal comments up

* Apply suggestions from code review

Co-Authored-By: LB <laurie@gatsbyjs.com>

* code suggestions

* more review suggestions

Co-authored-by: LB <laurie@gatsbyjs.com>

Author: gillkyle

docs/docs/creating-a-source-plugin.md

@@ -102,6 +102,8 @@ Now you have a `File` node to work with:
 
 Now, transform the newly created `File` nodes by hooking into the `onCreateNode` API in `gatsby-node.js`.
 
+#### Convert yaml into JSON for storage in Gatsby nodes
+
 If you're following along in an example project, install the following packages:
 
 ```shell
@@ -137,41 +139,67 @@ File content:
 
 Parsed YAML content:
 
-```javascript
-;[
+```json
+[
   {
-    id: "Jane Doe",
-    bio: "Developer based in Somewhere, USA",
+    "id": "Jane Doe",
+    "bio": "Developer based in Somewhere, USA"
   },
   {
-    id: "John Smith",
-    bio: "Developer based in Maintown, USA",
-  },
+    "id": "John Smith",
+    "bio": "Developer based in Maintown, USA"
+  }
 ]
 ```
 
 Now you'll write a helper function to transform the parsed YAML content into new Gatsby nodes:
 
-```javascript
+```javascript:title=gatsby-node.js
 function transformObject(obj, id, type) {
   const yamlNode = {
     ...obj,
     id,
     children: [],
-    parent: node.id,
+    parent: null,
     internal: {
       contentDigest: createContentDigest(obj),
       type,
     },
   }
   createNode(yamlNode)
-  createParentChildLink({ parent: node, child: yamlNode })
 }
 ```
 
 Above, you create a `yamlNode` object with the shape expected by the [`createNode` action](/docs/actions/#createNode).
 
-You then create a link between the parent node (file) and the child node (yaml content).
+#### Creating the transformer relationship
+
+You then need to create a link between the parent node (file) and the child node (yaml content) using the `createParentChildLink` function after adding the parent node's id to the `yamlNode`:
+
+```javascript:title=gatsby-node.js
+function transformObject(obj, id, type) {
+  const yamlNode = {
+    ...obj,
+    id,
+    children: [],
+    parent: node.id, // highlight-line
+    internal: {
+      contentDigest: createContentDigest(obj),
+      type,
+    },
+  }
+  createNode(yamlNode)
+  createParentChildLink({ parent: node, child: yamlNode }) // highlight-line
+}
+```
+
+Another example of a transformation relationship is the `gatsby-source-filesystem` plugin used with the `gatsby-transformer-remark` plugin. This combination transforms a parent `File` node's markdown string into a `MarkdownRemark` node. The remark transformer plugin adds its newly created child node as a child of the parent node using the action [`createParentChildLink`](/docs/actions/#createParentChildLink). Transformation relationships like this are used when a new node is _completely_ derived from a single parent node. E.g. the markdown node is derived from the parent `File` node and would not exist if the parent `File` node hadn't been created.
+
+Because all children nodes are derived from their parent, when a parent node is deleted or changed, Gatsby deletes all of the child nodes (and their child nodes, and so on). Gatsby does so with the expectation that they'll be recreated again by transformer plugins. This is done to ensure there are no nodes left over that were derived from older versions of data but should no longer exist.
+
+_For examples of other plugins creating transformation relationships, you can see the [`gatsby-transformer-remark` plugin](https://github.com/gatsbyjs/gatsby/blob/72077527b4acd3f2109ed5a2fcb780cddefee35a/packages/gatsby-transformer-remark/src/on-node-create.js#L39-L67) (from the above example) or the [`gatsby-transformer-sharp` plugin](https://github.com/gatsbyjs/gatsby/blob/1fb19f9ad16618acdac7eda33d295d8ceba7f393/packages/gatsby-transformer-sharp/src/on-node-create.js#L3-L25)._
+
+#### Create new nodes from the derived data
 
 In your updated `gatsby-node.js`, you'll then iterate through the parsed YAML content, using the helper function to transform each into a new node:
 
@@ -227,6 +255,8 @@ async function onCreateNode({
 exports.onCreateNode = onCreateNode
 ```
 
+#### Query for the transformed data
+
 Now you can query for your new nodes containing our transformed YAML data:
 
 ```graphql

docs/docs/creating-a-transformer-plugin.md

@@ -0,0 +1,83 @@
+# Creating First Class Gatsby Source Plugins
+
+Create Gatsby plugins that leverage Gatsby's most impactful native features like remote image optimization, caching, customized GraphQL schemas and node relationships, and more.
+
+This monorepo serves as an example of a site using a first class source plugin to pull in data from a Node.js API. It is meant to show the 3 pieces that work together when building a source plugin: the API, the site, and the source plugin.
+
+## Setup
+
+This monorepo uses yarn workspaces to manage the 3 indivdual projects:
+
+- api: a Node.js API with in-memory data, and a Post and Author type, as well as support for subscriptions when Posts are mutated
+- example-site: a barebones Gatsby site that implements the source plugin
+- source-plugin: a plugin that uses several Gatsby APIs to source data from the API, create responsive/optimized images from remote locations, and link the nodes in the example site
+
+To install dependencies for all projects run the install command in the root of the yarn workspace (which requires yarn to be installed):
+
+```
+yarn install
+```
+
+_Note: if you aren't using yarn, you can navigate into each of the 3 folders and run `npm install` instead_
+
+Then you can run the api or example projects in separate terminal windows with the commands below.
+
+For the API which runs at `localhost:4000`, use this command:
+
+```
+yarn workspace api start
+```
+
+And to run the example site with `gatsby develop` at `localhost:8000`, use this command:
+
+```
+yarn workspace example-site develop
+```
+
+Running the example site also runs the plugin because it is included in the site's config. You'll see output in the console for different functionality and then can open up the browser to `localhost:8000` to see the site.
+
+## Developing and Experimenting
+
+You can open up `localhost:4000` with the API running, which will load a GraphQL Playground, which is a GraphQL IDE (like GraphiQL, that Gatsby runs at `localhost:8000/___graphql`) for running queries and mutations on the data from the API.
+
+You can test a query like this to see data returned:
+
+```graphql
+query {
+  posts {
+    id
+    slug
+  }
+}
+```
+
+This query will return the IDs for all posts in the API. You can copy one of these IDs and provide it as an argument to a mutation to update information about that post.
+
+You can run 3 different mutations from the GraphQL Playground (at `localhost:4000`): `createPost`, `updatePost`, and `deletePost`. These methods would mimic CRUD operations happening on the API of the data source like a headless CMS. An example `updatePost` mutation is outlined below.
+
+When you run a mutation on a post, a subscription event is published, which lets the plugin know it should respond and update nodes. The following mutation can be copied into the left side of the GraphQL playground so long as you replace "post-id" with a value returned for an ID from a query (like the one above).
+
+```graphql
+mutation {
+  updatePost(id: "post-id", description: "Some data!") {
+    id
+    slug
+    description
+  }
+}
+```
+
+The website's homepage will update with any changes while the source plugin is subscribed to changes, which is when the `preview: true` is provided in the example site's `gatsby-config`.
+
+You can also optionally listen for subscription events with this query in the playground which will display data when a mutation is run:
+
+```graphql
+subscription {
+  posts {
+    id
+    description
+  }
+}
+```
+
+A similar subscription is registered when the plugin is run, so you can also see subscription events logged when the plugin is running.

examples/creating-source-plugins/README.md

@@ -0,0 +1,3 @@
+# Example API
+
+A small GraphQL server with in-memory data, powered by [graphql-yoga](https://github.com/graphcool/graphql-yoga) 🧘. See the root of the monorepo for details about running this API alongisde the `example-site` and `source-plugin`.

examples/creating-source-plugins/api/README.md

@@ -0,0 +1,43 @@
+{
+  "name": "api",
+  "description": "A simple GraphQL server example with in-memory data",
+  "version": "1.0.0",
+  "license": "MIT",
+  "homepage": "https://general-repair.glitch.me",
+  "author": {
+    "name": "Risan Bagja Pradana",
+    "email": "risanbagja@gmail.com",
+    "url": "https://risan.io"
+  },
+  "main": "src/index.js",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/risan/simple-graphql-server-example.git"
+  },
+  "bugs": {
+    "url": "https://github.com/risan/simple-graphql-server-example/issues"
+  },
+  "keywords": [
+    "graphql",
+    "graphql-server",
+    "graphql-yoga"
+  ],
+  "scripts": {
+    "start": "node src/index.js",
+    "lint": "eslint *.js src",
+    "lint-fix": "eslint *.js src --fix"
+  },
+  "dependencies": {
+    "dotenv": "^5.0.1",
+    "graphql-yoga": "^1.8.2",
+    "uniqid": "^4.1.1"
+  },
+  "devDependencies": {
+    "eslint": "^4.19.1",
+    "eslint-config-airbnb-base": "^12.1.0",
+    "eslint-config-prettier": "^2.9.0",
+    "eslint-plugin-import": "^2.10.0",
+    "eslint-plugin-prettier": "^2.6.0",
+    "prettier": "^1.11.1"
+  }
+}

examples/creating-source-plugins/api/package.json

@@ -0,0 +1,138 @@
+require("dotenv").config()
+const { GraphQLServer, PubSub } = require("graphql-yoga")
+const uniqid = require("uniqid")
+
+const CREATED = "created"
+const UPDATED = "updated"
+const DELETED = "deleted"
+
+const authors = [
+  {
+    id: 1,
+    name: "Jay Gatsby",
+  },
+  {
+    id: 2,
+    name: "Daisy Buchanan",
+  },
+]
+
+const posts = [
+  {
+    id: uniqid(),
+    slug: "hello-world",
+    description: "Our first post on our site.",
+    imgUrl: "https://images.unsplash.com/photo-1534432586043-ead5b99229fb",
+    imgAlt: "Pug in a sweater",
+    authorId: 1,
+  },
+  {
+    id: uniqid(),
+    slug: "company-vision",
+    description: "Our vision for a welcoming company.",
+    imgUrl: "https://images.unsplash.com/photo-1530041539828-114de669390e",
+    imgAlt: "Pug in a rainjacket",
+    authorId: 1,
+  },
+  {
+    id: uniqid(),
+    slug: "redesigning-our-logo",
+    description: "What went into the new logo.",
+    imgUrl: "https://images.unsplash.com/photo-1541364983171-a8ba01e95cfc",
+    imgAlt: "Pug in glasses",
+    authorId: 2,
+  },
+]
+
+const resolvers = {
+  Query: {
+    info: () => "A simple GraphQL server example with in-memory data.",
+    posts: () => posts,
+    authors: () => authors,
+  },
+
+  Mutation: {
+    createPost: (root, { slug, description }) => {
+      const post = {
+        id: uniqid(),
+        slug,
+        description,
+        imgUrl: "https://images.unsplash.com/photo-1534432586043-ead5b99229fb",
+        imgAlt: "pug in a sweater",
+        authorId: 1,
+      }
+
+      posts.push(post)
+      pubsub.publish(CREATED, { posts: [{ status: CREATED, ...post }] })
+
+      return post
+    },
+
+    updatePost: (root, { id, description }) => {
+      const postIdx = posts.findIndex(p => id === p.id)
+
+      if (postIdx === null) {
+        return null
+      }
+
+      posts[postIdx] = { ...posts[postIdx], description }
+      pubsub.publish(UPDATED, {
+        posts: [{ status: UPDATED, ...posts[postIdx] }],
+      })
+
+      return posts[postIdx]
+    },
+
+    deletePost: (root, { id }) => {
+      const postIdx = posts.findIndex(p => id === p.id)
+
+      if (postIdx === null) {
+        return null
+      }
+
+      const post = posts[postIdx]
+      pubsub.publish(DELETED, {
+        posts: [{ status: DELETED, ...posts[postIdx] }],
+      })
+
+      posts.splice(postIdx, 1)
+
+      return post
+    },
+  },
+
+  Post: {
+    id: root => root.id,
+    slug: root => root.slug,
+    description: root => root.description,
+    author: root => authors.find(author => author.id === root.authorId),
+  },
+
+  Author: {
+    id: root => root.id,
+    name: root => root.name,
+  },
+
+  Subscription: {
+    posts: {
+      subscribe: (parent, args, { pubsub }) => {
+        return pubsub.asyncIterator([CREATED, UPDATED, DELETED])
+      },
+    },
+  },
+}
+
+const pubsub = new PubSub()
+const server = new GraphQLServer({
+  typeDefs: "./src/schema.graphql",
+  resolvers,
+  context: { pubsub },
+})
+
+server.start(
+  {
+    port:
+      (process.env.PORT ? parseInt(process.env.PORT, 10) : undefined) || 4000,
+  },
+  ({ port }) => console.log(`🏃🏻‍ Server is running on port ${port}.`)
+)

examples/creating-source-plugins/api/src/index.js

@@ -0,0 +1,30 @@
+type Query {
+  info: String!
+  posts: [Post!]!
+  authors: [Author!]!
+}
+
+type Mutation {
+  createPost(slug: String!, description: String!): Post!
+  updatePost(id: ID!, description: String!): Post
+  deletePost(id: ID!): Post
+}
+
+type Post {
+  id: ID!
+  slug: String!
+  description: String!
+  imgUrl: String!
+  imgAlt: String!
+  author: Author!
+  status: String
+}
+
+type Author {
+  id: ID!
+  name: String!
+}
+
+type Subscription {
+  posts: [Post!]!
+}