feat(gatsby): SSR pages during development (#27432)

* Prototype SSR wip * Make it work * Fix issue with running two instances of webpack * Add a test suite for SSR * Linting * Run tests in CI * Lint * Show activity for HTML rendering + renable socket.io so server doesn't crash * Add error page when templates don't render correctly * Rebuild dev ssr bundle when source files change * Fix some lint errors * Fix building html * use gatsby colors for syntax highlighting * Add test script to compare dev & prod output * Add return types * Don't respond to dev server page loads until any sourcing/transforming is done * typings * fix types... maybe * maybe typescript happy * Add missing globals to tests & update snapshots * moer merry type work * Remove outdated typography.js test * Start migrating route handler to spawned service * back out moving dev html route into state machine * cleanups * more cleanups * yet moer cleanups * Add test for error parsing & codeframe creation * add return type * Feature flag functionality behind env variable * Lighten how much the dev ssr html webpack instance is watching * cleanup * track usage while we're in experimental stage * update snapshots * cleanup * Restore support for dev 404 page * fix test * Catch reading errors * Make static queries work in dev-ssr * Keep the renderer around * Send 'loading' page if webpack is busy * Show more error so can debug CI * lazily build dev 404 page so it happens after webpack/queries are run * just use / * Add ansi-html as depdnency * Run test w/ experimental flag * meaningless change to run tests again * use older version compatable with CI * Update packages/gatsby/cache-dir/develop-static-entry.js Co-authored-by: Peter van der Zee <209817+pvdz@users.noreply.github.com> * remove unnecessary changes * Consistently use imports * fix * Conditionally generate body str * remove unneeded change * Add find-page-by-path util from @pieh * make typescript happy * Switch to use @pieh's page finder util * fix find-page-by-path tests * Enable dev ssr for tests * Add build:types again * Do not await a flush * Only delete the render-page.js module cache when it changes * Try to reduce memory retention * Fix recreating dev 404 page on every request + cache requires * Add return * this wasn't necessary * Remove unused var * fix return type * Share cache across develop/develop-html instances of webpack * This caused a lot of runtime tests to fail * Use the webpack hash * This didn't work * fix lint error * Meaningless change to try tests again * SSR pages in jest-worker so memory doesn't accumulate in main process * fix lint * make typescript happy too * fix test import * Automatically fork the dev ssr renderer so it's ready to go when the user requests a page * Add structured logging on dev ssr failure * Need require.resolve I think * Add filepath + line/column to terminal error * try try again * Fixes hopefully * typescript 😱 * lint * Try tweaking jest settings * Debuggin * use default reporter * explicitly init dev html worker pool so it doesn't start during tests * restore original ci settings * sup * Update packages/gatsby-cli/src/structured-errors/error-map.ts Co-authored-by: Lennart <lekoarts@gmail.com> * console.logs seem to break jest-worker on CI * Increase pageLoadTimeout * try taskTimeout * This might be confusing cypress * Don't re-spawn the worker process on every change as that's very expensive. Just delete the module cache for 25 edits before re-spawning * cleanups * Update packages/gatsby/cache-dir/develop-static-entry.js Co-authored-by: Ward Peeters <ward@coding-tech.com> * Cleanups suggested by @wardpeet * fix lint * Lazily compile page components This makes the initial creation of the dev ssr bundle ~85% faster. * fix typescript * mock /lazy-sync-requires * The lazy bundling created a race condition where two pages could be simultaneously requested but both would think they're done as soon as the first to arrive finishes — 'suspend' rendering until the pageComponent is found to avoid this * Add more pages to make sure we're going to hit the race condition * Check file directly that the page component has been added This is a lot simpler & more reliable * for some reason this lets log warnings from React not break jest-worker * fix test & comment * We can't use the gatsby reporter inside a child as it uses process.send for console.* which breaks jest-worker * Move writing lazyComponents to requires-writer & still use old develop-static-entry if no flag * update tests * Don't render body of pages w/ matchPath * use core util joinPath so works on windows * try again * try try again * remove mistakenly added file * fix pnp test * Move lazy bundling changes to #27932 * More cleanup * more cleanups * Test both old & new develop-static-entry * fix lint * more lint fixes * Use execa * fix tests * Update names * Fail more gracefully when we source-maps don't work * fix test & remove testing code * Update snapshot * fix dependency check * remove unused import * Update packages/gatsby/src/utils/dev-ssr/develop-html-route.ts Co-authored-by: Ward Peeters <ward@coding-tech.com> * revert unnecessary change * restore more old behavior + move all new requires behind the flag * fix * fix problem w/ merge w/ master Co-authored-by: Sidhartha Chatterjee <me@sidharthachatterjee.com> Co-authored-by: gatsbybot <mathews.kyle+gatsbybot@gmail.com> Co-authored-by: Peter van der Zee <209817+pvdz@users.noreply.github.com> Co-authored-by: Michal Piechowiak <misiek.piechowiak@gmail.com> Co-authored-by: Lennart <lekoarts@gmail.com> Co-authored-by: Ward Peeters <ward@coding-tech.com>
gatsbyjs · Nov 17, 2020 · 23da2c3 · 23da2c3
1 parent 6858f22
commit 23da2c3
Show file tree

Hide file tree

Showing 40 changed files with 1,390 additions and 113 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -271,6 +271,12 @@ jobs:
       - e2e-test:
           test_path: integration-tests/artifacts
 
+  integration_tests_ssr:
+    executor: node
+    steps:
+      - e2e-test:
+          test_path: integration-tests/ssr
+
   e2e_tests_path-prefix:
     <<: *e2e-executor
     environment:
@@ -582,6 +588,8 @@ workflows:
           <<: *e2e-test-workflow
       - integration_tests_artifacts:
           <<: *e2e-test-workflow
+      - integration_tests_ssr:
+          <<: *e2e-test-workflow
       - integration_tests_gatsby_cli:
           requires:
             - bootstrap

diff --git a/integration-tests/ssr/LICENSE b/integration-tests/ssr/LICENSE
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2018 gatsbyjs
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
diff --git a/integration-tests/ssr/README.md b/integration-tests/ssr/README.md
@@ -0,0 +1 @@
+## SSR test suite
diff --git a/integration-tests/ssr/__tests__/__snapshots__/ssr.js.snap b/integration-tests/ssr/__tests__/__snapshots__/ssr.js.snap
@@ -0,0 +1,20 @@
+// Jest Snapshot v1, https://goo.gl/fbAQLP
+
+exports[`SSR is run for a page when it is requested 1`] = `"<!DOCTYPE html><html><head><meta charSet=\\"utf-8\\"/><meta http-equiv=\\"x-ua-compatible\\" content=\\"ie=edge\\"/><meta name=\\"viewport\\" content=\\"width=device-width, initial-scale=1, shrink-to-fit=no\\"/><meta name=\\"note\\" content=\\"environment=development\\"/><script src=\\"/socket.io/socket.io.js\\"></script></head><body><div id=\\"___gatsby\\"><div style=\\"outline:none\\" tabindex=\\"-1\\" id=\\"gatsby-focus-wrapper\\"><div>Hello world</div></div><div id=\\"gatsby-announcer\\" style=\\"position:absolute;top:0;width:1px;height:1px;padding:0;overflow:hidden;clip:rect(0, 0, 0, 0);white-space:nowrap;border:0\\" aria-live=\\"assertive\\" aria-atomic=\\"true\\"></div></div><script src=\\"/polyfill.js\\" nomodule=\\"\\"></script><script src=\\"/commons.js\\"></script></body></html>"`;
+
+exports[`SSR it generates an error page correctly 1`] = `
+"<title>Develop SSR Error</title><h1>Error<h1>
+        <h2>The page didn't SSR correctly</h2>
+        <ul>
+          <li><strong>URL path:</strong> <code>/bad-page/</code></li>
+          <li><strong>File path:</strong> <code>src/pages/bad-page.js</code></li>
+        </ul>
+        <h3>error message</h3>
+        <p><code>window is not defined</code></p><pre style=\\"background:#fdfaf6;padding:8px;\\"><span style=\\"font-weight:normal;opacity:1;color:#452475;background:#fdfaf6;\\"> <span style=\\"color:#527713;\\"> 2 | </span></span>
+<span style=\\"font-weight:normal;opacity:1;color:#452475;background:#fdfaf6;\\"> <span style=\\"color:#527713;\\"> 3 | </span><span style=\\"color:#006500;\\">const</span> <span style=\\"color:#DB3A00;\\">Component</span> <span style=\\"color:#DB3A00;\\">=</span> () <span style=\\"color:#DB3A00;\\">=></span> {</span>
+<span style=\\"font-weight:normal;opacity:1;color:#452475;background:#fdfaf6;\\"><span style=\\"color:#096fb3;\\"><span style=\\"font-weight:bold;\\">></span></span><span style=\\"color:#527713;\\"> 4 | </span>  <span style=\\"color:#006500;\\">const</span> a <span style=\\"color:#DB3A00;\\">=</span> window<span style=\\"color:#DB3A00;\\">.</span>width</span>
+<span style=\\"font-weight:normal;opacity:1;color:#452475;background:#fdfaf6;\\"> <span style=\\"color:#527713;\\">   | </span>            <span style=\\"color:#096fb3;\\"><span style=\\"font-weight:bold;\\">^</span></span></span>
+<span style=\\"font-weight:normal;opacity:1;color:#452475;background:#fdfaf6;\\"> <span style=\\"color:#527713;\\"> 5 | </span></span>
+<span style=\\"font-weight:normal;opacity:1;color:#452475;background:#fdfaf6;\\"> <span style=\\"color:#527713;\\"> 6 | </span>  <span style=\\"color:#006500;\\">return</span> <span style=\\"color:#DB3A00;\\"><</span><span style=\\"color:#DB3A00;\\">div</span><span style=\\"color:#DB3A00;\\">></span>hi<span style=\\"color:#DB3A00;\\"><</span><span style=\\"color:#DB3A00;\\">/</span><span style=\\"color:#DB3A00;\\">div</span><span style=\\"color:#DB3A00;\\">></span></span>
+<span style=\\"font-weight:normal;opacity:1;color:#452475;background:#fdfaf6;\\"> <span style=\\"color:#527713;\\"> 7 | </span>}</span></pre>"
+`;
diff --git a/integration-tests/ssr/__tests__/fixtures/bad-page.js b/integration-tests/ssr/__tests__/fixtures/bad-page.js
@@ -0,0 +1,9 @@
+import React from "react"
+
+const Component = () => {
+  const a = window.width
+
+  return <div>hi</div>
+}
+
+export default Component
diff --git a/integration-tests/ssr/__tests__/ssr.js b/integration-tests/ssr/__tests__/ssr.js
@@ -0,0 +1,55 @@
+const fetch = require(`node-fetch`)
+const execa = require(`execa`)
+const fs = require(`fs-extra`)
+const path = require(`path`)
+
+describe(`SSR`, () => {
+  test(`is run for a page when it is requested`, async () => {
+    const html = await fetch(`http://localhost:8000/`).then(res => res.text())
+
+    expect(html).toMatchSnapshot()
+  })
+  test(`dev & build outputs match`, async () => {
+    const childProcess = await execa(`yarn`, [`test-output`])
+
+    expect(childProcess.code).toEqual(0)
+  })
+  test(`it generates an error page correctly`, async () => {
+    const src = path.join(__dirname, `/fixtures/bad-page.js`)
+    const dest = path.join(__dirname, `../src/pages/bad-page.js`)
+    fs.copySync(src, dest)
+
+    const pageUrl = `http://localhost:8000/bad-page/`
+    await new Promise(resolve => {
+      // Poll until the new page is bundled (so starts returning a non-404 status).
+      const testInterval = setInterval(() => {
+        fetch(pageUrl).then(res => {
+          if (res.status !== 404) {
+            clearInterval(testInterval)
+            resolve()
+          }
+        })
+      }, 1000)
+    })
+
+    const rawDevHtml = await fetch(
+      `http://localhost:8000/bad-page/`
+    ).then(res => res.text())
+    expect(rawDevHtml).toMatchSnapshot()
+    fs.remove(dest)
+
+    // After the page is gone, it'll 404.
+    await new Promise(resolve => {
+      setTimeout(() => {
+        const testInterval = setInterval(() => {
+          fetch(pageUrl).then(res => {
+            if (res.status === 404) {
+              clearInterval(testInterval)
+              resolve()
+            }
+          })
+        }, 400)
+      }, 400)
+    })
+  })
+})
diff --git a/integration-tests/ssr/gatsby-config.js b/integration-tests/ssr/gatsby-config.js
@@ -0,0 +1,10 @@
+module.exports = {
+  siteMetadata: {
+    title: `Hello world`,
+    author: `Sid Chatterjee`,
+    twitter: `chatsidhartha`,
+    github: `sidharthachatterjee`,
+    moreInfo: `Sid is amazing`,
+  },
+  plugins: [],
+}
diff --git a/integration-tests/ssr/jest.config.js b/integration-tests/ssr/jest.config.js
@@ -0,0 +1,3 @@
+module.exports = {
+  testPathIgnorePatterns: [`/node_modules/`, `__tests__/fixtures`, `.cache`],
+}
diff --git a/integration-tests/ssr/package.json b/integration-tests/ssr/package.json
@@ -0,0 +1,39 @@
+{
+  "name": "ssr",
+  "description": "A simplified bare-bones starter for Gatsby.",
+  "version": "0.1.0",
+  "author": "Sid Chatterjee",
+  "bugs": {
+    "url": "https://github.com/gatsbyjs/gatsby/issues"
+  },
+  "dependencies": {
+    "gatsby": "2.24.82-dev-1603131999086",
+    "react": "^16.12.0",
+    "react-dom": "^16.12.0"
+  },
+  "devDependencies": {
+    "cross-env": "^5.0.2",
+    "fs-extra": "^9.0.0",
+    "jest": "^24.0.0",
+    "jest-cli": "^24.0.0",
+    "jest-diff": "^24.0.0",
+    "npm-run-all": "4.1.5",
+    "start-server-and-test": "^1.11.3"
+  },
+  "license": "MIT",
+  "private": true,
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/gatsbyjs/gatsby-starter-hello-world"
+  },
+  "scripts": {
+    "build": "gatsby build",
+    "clean": "gatsby clean",
+    "develop": "gatsby develop",
+    "serve": "gatsby serve",
+    "start-dev-server": "start-server-and-test develop http://localhost:8000 test:jest",
+    "test": "cross-env GATSBY_EXPERIMENTAL_DEV_SSR=true npm-run-all -s build start-dev-server",
+    "test-output": "node test-output.js",
+    "test:jest": "jest --config=jest.config.js --runInBand"
+  }
+}
diff --git a/integration-tests/ssr/src/pages/hi.js b/integration-tests/ssr/src/pages/hi.js
@@ -0,0 +1,15 @@
+import React from "react"
+import { useStaticQuery, graphql } from "gatsby"
+
+export default function Inline() {
+  const { site } = useStaticQuery(graphql`
+    {
+      site {
+        siteMetadata {
+          title
+        }
+      }
+    }
+  `)
+  return <div>hi1 {site.siteMetadata.title}</div>
+}
diff --git a/integration-tests/ssr/src/pages/hi2.js b/integration-tests/ssr/src/pages/hi2.js
@@ -0,0 +1,15 @@
+import React from "react"
+import { useStaticQuery, graphql } from "gatsby"
+
+export default function Inline() {
+  const { site } = useStaticQuery(graphql`
+    {
+      site {
+        siteMetadata {
+          title
+        }
+      }
+    }
+  `)
+  return <div>hi2 {site.siteMetadata.title}</div>
+}
diff --git a/integration-tests/ssr/src/pages/hi3.js b/integration-tests/ssr/src/pages/hi3.js
@@ -0,0 +1,15 @@
+import React from "react"
+import { useStaticQuery, graphql } from "gatsby"
+
+export default function Inline() {
+  const { site } = useStaticQuery(graphql`
+    {
+      site {
+        siteMetadata {
+          title
+        }
+      }
+    }
+  `)
+  return <div> hi3{site.siteMetadata.title}</div>
+}
diff --git a/integration-tests/ssr/src/pages/index.js b/integration-tests/ssr/src/pages/index.js
@@ -0,0 +1,15 @@
+import React from "react"
+import { useStaticQuery, graphql } from "gatsby"
+
+export default function Inline() {
+  const { site } = useStaticQuery(graphql`
+    {
+      site {
+        siteMetadata {
+          title
+        }
+      }
+    }
+  `)
+  return <div>{site.siteMetadata.title}</div>
+}
diff --git a/integration-tests/ssr/test-output.js b/integration-tests/ssr/test-output.js
@@ -0,0 +1,96 @@
+// To run the test script manually on a site (e.g. to test a plugin):
+// - build the site first
+// - start the develop server
+// - run this script
+;(async function () {
+  const { getPageHtmlFilePath } = require(`gatsby/dist/utils/page-html`)
+  const { join } = require(`path`)
+  const fs = require(`fs-extra`)
+  const fetch = require(`node-fetch`)
+  const diff = require(`jest-diff`)
+  const prettier = require(`prettier`)
+  const cheerio = require(`cheerio`)
+  const stripAnsi = require(`strip-ansi`)
+
+  const devSiteBasePath = `http://localhost:8000`
+
+  const comparePath = async path => {
+    const format = htmlStr => prettier.format(htmlStr, { parser: `html` })
+
+    const filterHtml = htmlStr => {
+      const $ = cheerio.load(htmlStr)
+      // There are many script tag differences
+      $(`script`).remove()
+      // Only added in production. Dev uses css-loader
+      $(`#gatsby-global-css`).remove()
+      // Only in prod
+      $(`link[rel="preload"]`).remove()
+      // Only in prod
+      $(`meta[name="generator"]`).remove()
+      // Only in dev
+      $(`meta[name="note"]`).remove()
+
+      return $.html()
+    }
+
+    const builtHtml = format(
+      filterHtml(
+        fs.readFileSync(
+          getPageHtmlFilePath(join(process.cwd(), `public`), path),
+          `utf-8`
+        )
+      )
+    )
+
+    const rawDevHtml = await fetch(`${devSiteBasePath}/${path}`).then(res =>
+      res.text()
+    )
+
+    const devHtml = format(filterHtml(rawDevHtml))
+    const diffResult = diff(devHtml, builtHtml, {
+      contextLines: 3,
+      expand: false,
+    })
+    if (
+      stripAnsi(diffResult) === `Compared values have no visual difference.`
+    ) {
+      return true
+    } else {
+      console.log(`path "${path}" has differences between dev & prod`)
+      console.log(diffResult)
+      return false
+    }
+  }
+
+  const response = await fetch(`${devSiteBasePath}/__graphql`, {
+    method: `POST`,
+    headers: { "Content-Type": `application/json` },
+    body: JSON.stringify({
+      query: `query MyQuery {
+  allSitePage {
+    nodes {
+      path
+    }
+  }
+}
+`,
+    }),
+  }).then(res => res.json()) // expecting a json response
+
+  const paths = response.data.allSitePage.nodes
+    .map(n => n.path)
+    .filter(p => p !== `/dev-404-page/`)
+
+  console.log(
+    `testing these paths for differences between dev & prod outputs`,
+    paths
+  )
+
+  const results = await Promise.all(paths.map(p => comparePath(p)))
+  // Test all true
+  if (results.every(r => r)) {
+    process.exit(0)
+  } else {
+    process.exit(1)
+  }
+})()
diff --git a/packages/babel-plugin-remove-graphql-queries/src/index.ts b/packages/babel-plugin-remove-graphql-queries/src/index.ts
@@ -273,6 +273,7 @@ export default function ({ types: t }): PluginObj {
           JSXIdentifier(path2: NodePath<JSXIdentifier>): void {
             if (
               (process.env.NODE_ENV === `test` ||
+                state.opts.stage === `develop-html` ||
                 state.opts.stage === `build-html`) &&
               path2.isJSXIdentifier({ name: `StaticQuery` }) &&
               path2.referencesImport(`gatsby`, ``) &&
@@ -315,6 +316,7 @@ export default function ({ types: t }): PluginObj {
           CallExpression(path2: NodePath<CallExpression>): void {
             if (
               (process.env.NODE_ENV === `test` ||
+                state.opts.stage === `develop-html` ||
                 state.opts.stage === `build-html`) &&
               isUseStaticQuery(path2)
             ) {

diff --git a/packages/gatsby-cli/src/structured-errors/error-map.ts b/packages/gatsby-cli/src/structured-errors/error-map.ts
@@ -539,6 +539,20 @@ const errors = {
     level: Level.ERROR,
     docsUrl: `https://www.gatsbyjs.org/docs/gatsby-cli/#new`,
   },
+  "11614": {
+    text: ({
+      path,
+      filePath,
+      line,
+      column,
+    }): string => `The path "${path}" errored during SSR.
+
+    Edit its component ${filePath}${
+      line ? `:${line}:${column}` : ``
+    } to resolve the error.`,
+    level: Level.WARNING,
+    docsUrl: `https://gatsby.dev/debug-html`,
+  },
   // Watchdog
   "11701": {
     text: (context): string =>

diff --git a/packages/gatsby-plugin-typography/src/__tests__/gatsby-ssr.js b/packages/gatsby-plugin-typography/src/__tests__/gatsby-ssr.js
@@ -33,12 +33,6 @@ describe(`onRenderBody`, () => {
     ])
   })
 
-  it(`only invokes setHeadComponents if BUILD_STAGE is build-html`, () => {
-    const api = setup({}, `develop`)
-
-    expect(api.setHeadComponents).not.toHaveBeenCalled()
-  })
-
   it(`does not add google font if omitGoogleFont is passed`, () => {
     const api = setup({
       omitGoogleFont: true,