Add experimental wildcard remotePatterns config for upstream images (#36245)

## Description 
This PR implements a new configuration object in `next.config.js` called `experimental.images.remotePatterns`.

This will eventually deprecate `images.domains` because it covers the same use cases and more by allowing wildcard pattern matching on `hostname` and `pathname` and also allows restricting `protocol` and `port`.

## Feature

- [x] Implements an existing feature request.
- [x] Related issues linked
- [x] Unit tests added
- [x] Integration tests added
- [x] Documentation added
- [x] Telemetry added. In case of a feature if it's used or not.
- [x] Errors have helpful link attached, see `contributing.md`

## Related 

- Fixes #27925 
- Closes #18429 
- Closes #18632
- Closes #18730
- Closes #27345

Author: styfle

docs/api-reference/next/image.md

@@ -16,6 +16,7 @@ description: Enable Image Optimization with the built-in Image component.
 
 | Version   | Changes                                                                                               |
 | --------- | ----------------------------------------------------------------------------------------------------- |
+| `v12.1.7` | Experimental `remotePatterns` configuration added.                                                    |
 | `v12.1.1` | `style` prop added. Experimental[\*](#experimental-raw-layout-mode) support for `layout="raw"` added. |
 | `v12.1.0` | `dangerouslyAllowSVG` and `contentSecurityPolicy` configuration added.                                |
 | `v12.0.9` | `lazyRoot` prop added.                                                                                |
@@ -313,9 +314,64 @@ Other properties on the `<Image />` component will be passed to the underlying
 
 ## Configuration Options
 
+### Remote Patterns
+
+> Note: The `remotePatterns` configuration is currently **experimental** and subject to change. Please use [`domains`](#domains) for production use cases.
+
+To protect your application from malicious users, configuration is required in order to use external images. This ensures that only external images from your account can be served from the Next.js Image Optimization API. These external images can be configured with the `remotePatterns` property in your `next.config.js` file, as shown below:
+
+```js
+module.exports = {
+  experimental: {
+    images: {
+      remotePatterns: [
+        {
+          protocol: 'https',
+          hostname: 'example.com',
+          port: '',
+          pathname: '/account123/**',
+        },
+      ],
+    },
+  },
+}
+```
+
+> Note: The example above will ensure the `src` property of `next/image` must start with `https://example.com/account123/`. Any other protocol, hostname, port, or unmatched path will respond with 400 Bad Request.
+
+Below is another example of the `remotePatterns` property in the `next.config.js` file:
+
+```js
+module.exports = {
+  experimental: {
+    images: {
+      remotePatterns: [
+        {
+          protocol: 'https',
+          hostname: '**.example.com',
+        },
+      ],
+    },
+  },
+}
+```
+
+> Note: The example above will ensure the `src` property of `next/image` must start with `https://img1.example.com` or `https://me.avatar.example.com` or any number of subdomains. Any other protocol or unmatched hostname will respond with 400 Bad Request.
+
+Wildcard patterns can be used for both `pathname` and `hostname` and have the following syntax:
+
+- `*` match a single path segment or subdomain
+- `**` match any number of path segments at the end or subdomains at the beginning
+
+The `**` syntax does not work in the middle of the pattern.
+
 ### Domains
 
-To protect your application from malicious users, you must define a list of image provider domains that you want to be served from the Next.js Image Optimization API. This is configured in with the `domains` property in your `next.config.js` file, as shown below:
+Similar to [`remotePatterns`](#remote-patterns), the `domains` configuration can be used to provide a list of allowed hostnames for external images.
+
+However, the `domains` configuration does not support wildcard pattern matching and it cannot restrict protocol, port, or pathname.
+
+Below is an example of the `domains` property in the `next.config.js` file:
 
 ```js
 module.exports = {

docs/basic-features/image-optimization.md

@@ -66,7 +66,7 @@ function Home() {
 
 ### Remote Images
 
-To use a remote image, the `src` property should be a URL string, which can be [relative](#loaders) or [absolute](#domains). Because Next.js does not have access to remote files during the build process, you'll need to provide the [`width`](/docs/api-reference/next/image.md#width), [`height`](/docs/api-reference/next/image.md#height) and optional [`blurDataURL`](/docs/api-reference/next/image.md#blurdataurl) props manually:
+To use a remote image, the `src` property should be a URL string, which can be [relative](#loaders) or [absolute](/docs/api-reference/next/image.md#domains). Because Next.js does not have access to remote files during the build process, you'll need to provide the [`width`](/docs/api-reference/next/image.md#width), [`height`](/docs/api-reference/next/image.md#height) and optional [`blurDataURL`](/docs/api-reference/next/image.md#blurdataurl) props manually:
 
 ```jsx
 import Image from 'next/image'
@@ -93,15 +93,9 @@ export default function Home() {
 
 Sometimes you may want to access a remote image, but still use the built-in Next.js Image Optimization API. To do this, leave the `loader` at its default setting and enter an absolute URL for the Image `src`.
 
-To protect your application from malicious users, you must define a list of remote domains that you intend to access this way. This is configured in your `next.config.js` file, as shown below:
+To protect your application from malicious users, you must define a list of remote hostnames you intend to allow remote access.
 
-```js
-module.exports = {
-  images: {
-    domains: ['example.com', 'example2.com'],
-  },
-}
-```
+> Learn more about [`domains`](/docs/api-reference/next/image.md#domains) configuration.
 
 ### Loaders
 
@@ -207,7 +201,7 @@ For examples of the Image component used with the various fill modes, see the [I
 
 ## Configuration
 
-The `next/image` component and Next.js Image Optimization API can be configured in the [`next.config.js` file](/docs/api-reference/next.config.js/introduction.md). These configurations allow you to [enable remote domains](/docs/api-reference/next/image.md#domains), [define custom image breakpoints](/docs/api-reference/next/image.md#device-sizes), [change caching behavior](/docs/api-reference/next/image.md#caching-behavior) and more.
+The `next/image` component and Next.js Image Optimization API can be configured in the [`next.config.js` file](/docs/api-reference/next.config.js/introduction.md). These configurations allow you to [enable remote images](/docs/api-reference/next/image.md#domains), [define custom image breakpoints](/docs/api-reference/next/image.md#device-sizes), [change caching behavior](/docs/api-reference/next/image.md#caching-behavior) and more.
 
 [**Read the full image configuration documentation for more information.**](/docs/api-reference/next/image.md#configuration-options)
 

errors/invalid-images-config.md

@@ -17,6 +17,8 @@ module.exports = {
     imageSizes: [16, 32, 48, 64, 96, 128, 256, 384],
     // limit of 50 domains values
     domains: [],
+    // limit of 50 objects
+    remotePatterns: [],
     // path prefix for Image Optimization API, useful with `loader`
     path: '/_next/image',
     // loader can be 'default', 'imgix', 'cloudinary', 'akamai', or 'custom'

packages/next/build/index.ts

@@ -2132,6 +2132,8 @@ export default async function build(
       const images = { ...config.images }
       const { deviceSizes, imageSizes } = images
       ;(images as any).sizes = [...deviceSizes, ...imageSizes]
+      ;(images as any).remotePatterns =
+        config?.experimental?.images?.remotePatterns || []
 
       await promises.writeFile(
         path.join(distDir, IMAGES_MANIFEST),

packages/next/build/webpack-config.ts

@@ -1467,6 +1467,8 @@ export default async function getBaseWebpackConfig(
             ? {
                 // pass domains in development to allow validating on the client
                 domains: config.images.domains,
+                experimentalRemotePatterns:
+                  config.experimental?.images?.remotePatterns,
               }
             : {}),
         }),

packages/next/client/image.tsx

@@ -18,8 +18,8 @@ import { ImageConfigContext } from '../shared/lib/image-config-context'
 import { warnOnce } from '../shared/lib/utils'
 import { normalizePathTrailingSlash } from './normalize-trailing-slash'
 
-const experimentalLayoutRaw = (process.env.__NEXT_IMAGE_OPTS as any)
-  ?.experimentalLayoutRaw
+const { experimentalLayoutRaw = false, experimentalRemotePatterns = [] } =
+  (process.env.__NEXT_IMAGE_OPTS as any) || {}
 const configEnv = process.env.__NEXT_IMAGE_OPTS as any as ImageConfigComplete
 const loadedImageURLs = new Set<string>()
 const allImgs = new Map<
@@ -1063,7 +1063,10 @@ function defaultLoader({
       )
     }
 
-    if (!src.startsWith('/') && config.domains) {
+    if (
+      !src.startsWith('/') &&
+      (config.domains || experimentalRemotePatterns)
+    ) {
       let parsedSrc: URL
       try {
         parsedSrc = new URL(src)
@@ -1074,14 +1077,15 @@ function defaultLoader({
         )
       }
 
-      if (
-        process.env.NODE_ENV !== 'test' &&
-        !config.domains.includes(parsedSrc.hostname)
-      ) {
-        throw new Error(
-          `Invalid src prop (${src}) on \`next/image\`, hostname "${parsedSrc.hostname}" is not configured under images in your \`next.config.js\`\n` +
-            `See more info: https://nextjs.org/docs/messages/next-image-unconfigured-host`
-        )
+      if (process.env.NODE_ENV !== 'test') {
+        // We use dynamic require because this should only error in development
+        const { hasMatch } = require('../shared/lib/match-remote-pattern')
+        if (!hasMatch(config.domains, experimentalRemotePatterns, parsedSrc)) {
+          throw new Error(
+            `Invalid src prop (${src}) on \`next/image\`, hostname "${parsedSrc.hostname}" is not configured under images in your \`next.config.js\`\n` +
+              `See more info: https://nextjs.org/docs/messages/next-image-unconfigured-host`
+          )
+        }
       }
     }
   }

packages/next/server/config-shared.ts

@@ -5,6 +5,7 @@ import {
   ImageConfig,
   ImageConfigComplete,
   imageConfigDefault,
+  RemotePattern,
 } from '../shared/lib/image-config'
 
 export type PageRuntime = 'nodejs' | 'edge' | undefined
@@ -115,6 +116,7 @@ export interface ExperimentalConfig {
   outputStandalone?: boolean
   images?: {
     layoutRaw: boolean
+    remotePatterns: RemotePattern[]
   }
   middlewareSourceMaps?: boolean
   emotion?:
@@ -501,6 +503,7 @@ export const defaultConfig: NextConfig = {
     outputStandalone: !!process.env.NEXT_PRIVATE_STANDALONE,
     images: {
       layoutRaw: false,
+      remotePatterns: [],
     },
   },
 }

packages/next/server/config.ts

@@ -234,6 +234,43 @@ function assignDefaults(userConfig: { [key: string]: any }) {
         )
       }
     }
+
+    const remotePatterns = result.experimental?.images?.remotePatterns
+    if (remotePatterns) {
+      if (!Array.isArray(remotePatterns)) {
+        throw new Error(
+          `Specified images.remotePatterns should be an Array received ${typeof remotePatterns}.\nSee more info here: https://nextjs.org/docs/messages/invalid-images-config`
+        )
+      }
+
+      if (remotePatterns.length > 50) {
+        throw new Error(
+          `Specified images.remotePatterns exceeds length of 50, received length (${remotePatterns.length}), please reduce the length of the array to continue.\nSee more info here: https://nextjs.org/docs/messages/invalid-images-config`
+        )
+      }
+
+      const validProps = new Set(['protocol', 'hostname', 'pathname', 'port'])
+      const requiredProps = ['hostname']
+      const invalidPatterns = remotePatterns.filter(
+        (d: unknown) =>
+          !d ||
+          typeof d !== 'object' ||
+          Object.entries(d).some(
+            ([k, v]) => !validProps.has(k) || typeof v !== 'string'
+          ) ||
+          requiredProps.some((k) => !(k in d))
+      )
+      if (invalidPatterns.length > 0) {
+        throw new Error(
+          `Invalid images.remotePatterns values:\n${invalidPatterns
+            .map((item) => JSON.stringify(item))
+            .join(
+              '\n'
+            )}\n\nremotePatterns value must follow format { protocol: 'https', hostname: 'example.com', port: '', pathname: '/imgs/**' }.\nSee more info here: https://nextjs.org/docs/messages/invalid-images-config`
+        )
+      }
+    }
+
     if (images.deviceSizes) {
       const { deviceSizes } = images
       if (!Array.isArray(deviceSizes)) {

packages/next/server/image-optimizer.ts

@@ -17,6 +17,7 @@ import chalk from 'next/dist/compiled/chalk'
 import { NextUrlWithParsedQuery } from './request-meta'
 import { IncrementalCacheEntry, IncrementalCacheValue } from './response-cache'
 import { mockRequest } from './lib/mock-request'
+import { hasMatch } from '../shared/lib/match-remote-pattern'
 
 type XCacheHeader = 'MISS' | 'HIT' | 'STALE'
 
@@ -75,6 +76,7 @@ export class ImageOptimizerCache {
       minimumCacheTTL = 60,
       formats = ['image/webp'],
     } = imageData
+    const remotePatterns = nextConfig.experimental.images?.remotePatterns || []
     const { url, w, q } = query
     let href: string
 
@@ -104,7 +106,7 @@ export class ImageOptimizerCache {
         return { errorMessage: '"url" parameter is invalid' }
       }
 
-      if (!domains || !domains.includes(hrefParsed.hostname)) {
+      if (!hasMatch(domains, remotePatterns, hrefParsed)) {
         return { errorMessage: '"url" parameter is not allowed' }
       }
     }

packages/next/shared/lib/image-config.ts

@@ -8,6 +8,33 @@ export const VALID_LOADERS = [
 
 export type LoaderValue = typeof VALID_LOADERS[number]
 
+export type RemotePattern = {
+  /**
+   * Must be `http` or `https`.
+   */
+  protocol?: 'http' | 'https'
+
+  /**
+   * Can be literal or wildcard.
+   * Single `*` matches a single subdomain.
+   * Double `**` matches any number of subdomains.
+   */
+  hostname: string
+
+  /**
+   * Can be literal port such as `8080` or empty string
+   * meaning no port.
+   */
+  port?: string
+
+  /**
+   * Can be literal or wildcard.
+   * Single `*` matches a single path segment.
+   * Double `**` matches any number of path segments.
+   */
+  pathname?: string
+}
+
 type ImageFormat = 'image/avif' | 'image/webp'
 
 /**
@@ -28,7 +55,9 @@ export type ImageConfigComplete = {
   /** @see [Image loader configuration](https://nextjs.org/docs/api-reference/next/image#loader-configuration) */
   path: string
 
-  /** @see [Image domains configuration](https://nextjs.org/docs/basic-features/image-optimization#domains) */
+  /**
+   * @see [Image domains configuration](https://nextjs.org/docs/api-reference/next/image#domains)
+   */
   domains: string[]
 
   /** @see [Cache behavior](https://nextjs.org/docs/api-reference/next/image#caching-behavior) */