Experience Hostname rewrite (and why still no Vercel... 😩)

[samfromaway]

As discussed with @lfades , I wanted to share my learnings for this complex project with the community, as I think it's always interesting in something like _middlware that is still in beta, when it's used in an actual complex project, as I found some things not 100% polished yet.

With the help of @steven-tey I found this example https://github.com/vercel/examples/tree/main/edge-functions/hostname-rewrites to do:

The project
In our company we use a multi-brand website platform for our travel-agency partners.
Some of them use a multi-language approach (e.g. brand.com/de and some have just a single language page (e.g. brand.de). Also each brand has their own settings, colors and content.

Our problem
We have not a scalable approach of setting up our websites with our different brands, which lead to infrastructure issues (crashing servers) when building the websites in our GCP (google cloud platform self hosting).

Goal
Reduce multiple envs/builds for each brand (30), to just 1 env/build, where the project is mapped not by the individual env-variables but the domain.

Challanges:

• How to internationalize the content
• How to do the internationalized routing for both multi-lang and single-lang in one project
• How to do it statically
• How to map redirections we set in an admin (json file)

Our basic data flow in the project

• We use getInitialProps in the _app to pass the global brand settings
• We pass props in getStaticProps for the page content
• We use next-i18n-next to map the translation keys with the actual content which is diferent by brand
• We map a json where we define our redirects and use redirects in next.config.js
• We rewrite the routes if single-language-page from e.g. "/viajes" to our folder name "/trips" in next.config.js

Refactoring
Using the _sites/[site] approach of the example mentioned before, I had to do this:

• Change the rewrites from next.config.js to _middleware
• Change the redirects from next.config.js to _middleware
• Add getStaticPaths to all pages
• Remove identifiers like brand_id from public-runtime-config to params.site

Issues along the way

• I had to get rid of next-i18n-next as we got issues with the custom 404 and 500 page, as well as an issue that only the keys were loaded of the first brand that got a page-visit for all of the brands-websites instead of each brand having the corresponding content. I integrated my own mapping hook and context.

• public-runtime-config could be used in each file just importing it, I had to do a lot of prop drilling for the brand identifiers like brand_id because it comes from getStaticProps only

• Some TS error in _middleware, more down below

• Issue with redirecting pages that have .html at the end, more down below

• The wildcard matching of the next.config.js redirects did not work in _middleware, so I had to write my own wildcard matcher

• Issue with a page that used params, more down below

• We couldn't go to just one env, as next config needs the default-language for the i18n routing. So now we have 3 envs. (In the beginning it was 30)

Basic Explanation of my _middleware

1. pathname.startsWith(/_sites) return 404
2. !pathname.includes('.') && !pathname.startsWith('/api') || pathname.includes('.html') then do:
3. fetch redirects json
4. See if there is a redirect, if yes redirect
5. else get folder name from pathname depending on if is multi-lang, or single-lang-page
6. rewrite the route from e.g. pathname "/viajes" to folder name "trips" and adding the sitename like new URL(/_sites/${currentHost}/${folderName}, req.url)

Possible actionable from next.js

1. I get this TS error "Argument of type 'URL' is not assignable to parameter of type 'string | NextURL" when doing new URL(/_sites/${currentHost}/${folderName}, req.url)

2. If only if(!pathname.includes('.')) is added, pages with .html will not be handled correctly. I added an additional "|| pathname.includes('.html')" then it worked

3. One of our pages uses url-params (e.g. ?startDate=2022) for fetching data in the getServerSideProps. This did not work in _middleware as I got an error. So this is the only page I have that is not under _sites[site} and is still geting the rewrite from next.config.js

Why not vercel
Since I started in the company I tried to convince the CTO to move to vercel. This was never possible because we had so many envs. So now we would have the option to switch, why did we not do it at the end:

• There was an issue with a filter-page where we use query params client side to set the filters. It works in our GCP but not on vercel.
• It was easier to convince stakeholders to stay instead of moving to vercel
• When navigating to brand.com/de, the website still showed in English like it was on brand.com. In our GCP this was not an issue.

I would have loved to switch though as the build time in our GCP is 25min and in vercel 3min...

All in all it was a more or less smooth experience, the Jest refactoring, e2e refactoring... all worked fine as well.

Packages used
Apollo
Next js
next-i18n-next (not anymore, now is a custom hook and context)
MUI
...

Hope it helps. Let me know if you have questions.
I will soon post a video on my youtube channel about it https://www.youtube.com/c/DevWorldInfo if you want more info.

Some example websites
https://www.galapatours.com/de/trips
https://www.viventura.de/reisen
https://www.viventura.fr/circuits

[lfades]

Hi @samfromaway thank you for leaving all this feedback 🙌

On "Our basic data flow in the project"

We use getInitialProps in the _app to pass the global brand settings
We pass props in getStaticProps for the page content

If you have getInitialProps in _app everything is doing SSR, even if you use getStaticProps, that can be tested by loading the example websites.

We use next-i18n-next to map the translation keys with the actual content which is diferent by brand
We map a json where we define our redirects and use redirects in next.config.js
We rewrite the routes if single-language-page from e.g. "/viajes" to our folder name "/trips" in next.config.js

This is all fine 👍

On "Refactoring"

Change the rewrites from next.config.js to _middleware
Change the redirects from next.config.js to _middleware

Having rewrites and redirects in next.config.js is okay, it's not needed to move them to middleware unless you can't make them work in next.config.js.

Remove identifiers like brand_id from public-runtime-config to params.site

I guess by this you mean using the prop defined by the path instead? e.g /_sites/${currentHost}/${folderName}

On "Issues along the way"

public-runtime-config could be used in each file just importing it, I had to do a lot of prop drilling for the brand identifiers like brand_id because it comes from getStaticProps only.

You can use React.Context here 👍

The wildcard matching of the next.config.js redirects did not work in _middleware, so I had to write my own wildcard matcher

There's no wildcard matching in _middleware, so that's expected.

On "Basic Explanation of my _middleware"

fetch redirects json
See if there is a redirect, if yes redirect

This can be very slow, unless you're fetching from somewhere it always close to the selected edge.

On "Possible actionable from next.js"

I get this TS error "Argument of type 'URL' is not assignable to parameter of type 'string | NextURL" when doing new URL(/_sites/${currentHost}/${folderName}, req.url)

I checked and looks like it's only happening for rewrites, is that right? Thank you for the catch.

If only if(!pathname.includes('.')) is added, pages with .html will not be handled correctly. I added an additional "|| pathname.includes('.html')" then it worked

That makes sense, the first check would exclude .html as they start with ..

One of our pages uses url-params (e.g. ?startDate=2022) for fetching data in the getServerSideProps. This did not work in _middleware as I got an error. So this is the only page I have that is not under _sites[site} and is still geting the rewrite from next.config.js

Could you provide a minimal reproduction of this error, or expand a bit more on it 🤔

On why not Vercel

There was an issue with a filter-page where we use query params client side to set the filters. It works in our GCP but not on vercel.

Can you expand more here on why doesn't it work? setting query params client-side should work in Vercel.

When navigating to brand.com/de, the website still showed in English like it was on brand.com. In our GCP this was not an issue.

How does your i18n setup look like? are you using Next.js i18n alongside Edge middleware or having your own i18n implementation using edge middleware? either way more details here are welcome!

[samfromaway]

@lfades Thanks for you reply:

1: "If you have getInitialProps in _app everything is doing SSR, even if you use getStaticProps, that can be tested by loading the example websites." Do you have documentation about how _app works with static site generation if using getInitialProps? Because for me I still get the behaviour of ISR like I would expect, so it seems that it's static.

2: "fetch redirects json". Does this run on every user-request, or only when the page is rerendering itself with ISR?

3: "TS error "Argument of type 'U...", yes only rewrites

4: I'll work on some examples for the other issues in the coming days.

[samfromaway]

Query param issues: I added this: vercel/next.js#35647

[samfromaway]

Issue with Next.js i18n in next.config.js. I could not reproduce it with a basic example. It seems to work.

[lfades]

Do you have documentation about how _app works with static site generation if using getInitialProps? Because for me I still get the behaviour of ISR like I would expect, so it seems that it's static.

It's not static, it's currently not possible to have static pages when getInitialProps is used in pages/_app. I can test that by checking the network tab for the request made to any of these sites:

https://www.galapatours.com/de/trips
https://www.viventura.de/reisen
https://www.viventura.fr/circuits

In average the site loads to me anywhere between 500ms to 1.2s, I'm in Colombia and the server IP is in San Francisco, usually for that distance I expect a latency of 200-250ms max, if I see that it's taking way longer then it's doing something else. You should be able to see the usage of your serverless execution raising.

"fetch redirects json". Does this run on every user-request, or only when the page is rerendering itself with ISR?

I'll assume we're talking about Next.js Middleware here, my answer was this:

This can be very slow, unless you're fetching from somewhere it always close to the selected edge.

So if you're running Next.js Middleware in the Edge like Vercel does by default (next start && next build outside Vercel won't do that for you) then it's ideal to always do fetch operations to a source that's very close to your edges, i.e to a replicated db like redis, or otherwise read a local file with tons of redirects instead, we go through that in our redirects example with Upstash.

If you're not running Next.js Middleware in the edge I recommend to always go for a local file to avoid adding more latency to the request.

[samfromaway]

@lfades
Thanks again for the reply.

1. For me it seems to work with static files. Check out this repo:
   https://github.com/samfromaway/get-init-props-test

The items are connected to the getInitialProps of _app
and the names are connected to getStaticProps of index.js

As you can see with https://get-init-props-test.vercel.app/ the page is served statically. To test that further I even added a 5s delay in getInitialProps of _app.
Also when I change the data of the Items (from getInitialProps of _app) it will not update immediately, it will update after 30s like I would expect ISR to work, even though it's coming from getInitialProps.
Do you know why it works? For me it's still unclear of how _app (especially with getInitialProps) works in combination with getStaticProps and getServerSideProps. Is there any docs about this?

2. Thanks, at the moment we are happy things work more or less fast and are satisfied at the moment with a load time of around 1s. Our infrastructure is struggeling a lot at the moment, so I'm not surprised it takes ca. 800ms for the page to reach you. So I think it's you general infrastructure that takes so long, not because we don't serve the files statically. But we will work on it in the comming weeks. Then we will have a look at putting the requests on the edge...

[samfromaway]

Thanks again for the reply. For me it seems to work with static files. Check out this repo: https://github.com/samfromaway/get-init-props-test The items are connected to the getInitialProps of _app and the names are connected to getStaticProps of index.js As you can see with https://get-init-props-test.vercel.app/ the page is served statically. To test that further I even added a 5s delay in getInitialProps of _app. Also when I change the data of the Items (from getInitialProps of _app) it will not update immediately, it will update after 30s like I would expect ISR to work, even though it's coming from getInitialProps. Do you know why it works? For me it's still unclear of how _app (especially with getInitialProps) works in combination with getStaticProps and getServerSideProps. Is there any docs about this? Thanks, at the moment we are happy things work more or less fast and are satisfied at the moment with a load time of around 1s. Our infrastructure is struggeling a lot at the moment, so I'm not surprised it takes ca. 800ms for the page to reach you. So I think it's you general infrastructure that takes so long, not because we don't serve the files statically. But we will work on it in the comming weeks. Then we will have a look at putting the requests on the edge...

…

On Wed, Mar 30, 2022 at 10:36 PM Luis Alvarez D. ***@***.***> wrote: Do you have documentation about how _app works with static site generation if using getInitialProps? Because for me I still get the behaviour of ISR like I would expect, so it seems that it's static. It's not static, it's currently not possible to have static pages when getInitialProps is used in pages/_app. I can test that by checking the network tab for the request made to any of these sites: https://www.galapatours.com/de/trips https://www.viventura.de/reisen https://www.viventura.fr/circuits [image: image] <https://user-images.githubusercontent.com/4278345/160925008-f9bbb728-0a39-4e07-a3e3-6614e21a1375.png> In average the site loads to me anywhere between 500ms to 1.2s, I'm in Colombia and the server IP is in San Francisco, usually for that distance I expect a latency of 200-250ms max, if I see that it's taking way longer then it's doing something else. You should be able to see the usage of your serverless execution raising. "fetch redirects json". Does this run on every user-request, or only when the page is rerendering itself with ISR? I'll assume we're talking about Next.js Middleware here, my answer was this: This can be very slow, unless you're fetching from somewhere it always close to the selected edge. So if you're running Next.js Middleware in the Edge like Vercel does by default (next start && next build outside Vercel won't do that for you) then it's ideal to always do fetch operations to a source that's very close to your edges, i.e to a replicated db like redis, or otherwise read a local file with tons of redirects instead, we go through that in our redirects example with Upstash <https://github.com/vercel/examples/tree/main/edge-functions/redirects-upstash> . If you're not running Next.js Middleware in the edge I recommend to always go for a local file to avoid adding more latency to the request. — Reply to this email directly, view it on GitHub <#201 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOQANKONNFLT2XS5FQ6BSJDVCS3KZANCNFSM5RQKC3JQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[lfades]

@samfromaway You're right! getInitialProps doesn't turn pages with getStaticProps into serverless functions, only pages without it. the getInitialProps will be used for revalidations and that's why you see it running every 30 seconds.