Publishing to Netlify

[jules32]

Summer solstice with @samanthacsik

Current Netlify setup

~ https://app.netlify.com/sites/sad-williams-fa509f/settings/deploys~ On further inspection, it looks like openscapes.org deploys from hardcore-borg-58bfb4:

publishes from https://github.com/openscapes/website

Quarto publish

Following https://quarto.org/docs/publishing/netlify.html#publish-command:

quarto publish netlify created https://app.netlify.com/sites/gleaming-biscochitos-2c8aab/overview

wooo

Goodbye

Hello

[jules32]

Wooo regular expressions for redirects:

blog/(\w+)-(\w+)-(\w+)(.*)

And then pasted into Google Sheets.

(Which we learned from Practical Computing for Biologists)

[jules32]

We've just published our site with Quarto and not Blogdown! 🎉
The site is at https://openscapes.org, from the repo https://github.com/openscapes/website-new.

We did this ultimately via Publish from GitHub although we first tried via Quarto Publish Command

However, our redirects aren't working on netlify, and we've been troubleshooting for awhile.

The site is up and the 10 Redirect Rules have been processing:

But we get 404 errors – and not from the 404.qmd that we created. So something is going on.

Here are our redirects:

https://github.com/Openscapes/website-new/blob/main/_redirects

And we have a full line-up ready here:

https://github.com/Openscapes/website-new/blob/main/redirects-list

We actually have aliases is most webpage yaml front matter for blog posts (which is the vast majority of redirects) but that didn't work so we saw the tip in the redirects documentation to use the _redirects file for netlify

[cderv]

Hey @jules32 !

Just looking at this following your slack message. Here is what I noticed:

From the screenshot you shared, I had the internal netlify adresse : https://dreamy-fudge-e71879.netlify.app/

So I tried to check the 404.html page there: https://dreamy-fudge-e71879.netlify.app/404.html and it is working.

Same for non existing page: https://dreamy-fudge-e71879.netlify.app/notexisting

and same for one of your redirect: https://dreamy-fudge-e71879.netlify.app/approach-guide/

So I wonder: did you checked that openscapes.org is correctly using your last deploy of netlify ?
Because indeed none of the above is working from https://openscapes.org/ but working from your netlify deployed site.

Hope it helps

[jules32]

Hi @cderv, thank you so much!!

That was absolutely what was going on – in our testing with Netlify, our domain openscapes.org was still attached to a different Netlify deploy, not the current functioning one with redirects. I've fixed this now, and will clean up those old deploys.

Thank you so much for looking at this and helping out. We'll include this in the blog post about transitioning from blogdown too :)

Have a good day!

[jules32]

I've tested a few redirects that are aliases in specific blog posts, for example: https://openscapes.org/blog/2023-04-06-kyber/.

---
title: "Designing automated workflows, a chat  with Dr. Sean Kross"
aliases: 
  - https://www.openscapes.org/blog/2023/04/06/kyber/
---

It is not redirecting. I am rebuilding the whole site with blog/_metadata.yml set to freeze: false to see if that's how to build all the redirect pages. If that doesn't work, then I'll add all the https://github.com/Openscapes/website-new/blob/main/redirects-list to _redirects.

[samanthacsik]

Good news is that the 404 page is showing up now though! Got this when trying the kyber post alias (https://www.openscapes.org/blog/2023/04/06/kyber/)

[jules32]

I'm seeing that too @samanthacsik: the blogs are still not redirecting: https://openscapes.org/blog/2023/04/06/kyber/.

I wonder if there should be .html redirect files created in https://github.com/Openscapes/website-new/tree/main/_site/blog?

[jules32]

Further sleuthing: I also get a 404 when trying from the Netlify URL (which I renamed to dreamy-solstice:

https://dreamy-solstice-e71879.netlify.app/blog/2023/04/06/kyber/

[samanthacsik]

@jules32 maybe we try a netlify.toml file? I've seen it come up a few times in the resources we were looking through last night (e.g. https://docs.netlify.com/routing/redirects/#syntax-for-the-netlify-configuration-file) and now watching this youtube video (~5:20). It's more verbose, but also more explicit

[samanthacsik]

to test, I think we could try something like:

[[redirects]]
  from = "/old-path"
  to = "/new-path"
  status = 301

[[redirects]]
  from = "/another-old-path"
  to = "/another-new-path"
  status = 301

[jules32]

@samanthacsik that did it: https://github.com/Openscapes/website-new/blob/main/netlify.toml 🎉 !! woo!

[[redirects]]
  from = "/blog/2023/04/06/kyber/"
  to = "/blog/2023-04-06-kyber/"
  status = 301

Would you please be able to add the other 140 posts from the _redirects file?

[samanthacsik]

WOWOWOW!! Yes can do!

[jules32]

Thank you!

[jules32]

One other thing to note about redirects/publishing:

https://openscapes.org/code-of-conduct/ returns this:

Whereas https://openscapes.org/code-of-conduct without the trailing / returns this:

[samanthacsik]

Formatting list of redirects for netlify.toml file (specifically blog posts)

Using a _redirects file (detailed in the comments above) didn't produce working redirects, but a netlify.toml file seemed to do the trick! @jules32 and I moved all redirects into a separate file called redirects-list for future fiddling. It looks like this (only a small subset of redirects shown here):

# Redirects from what the browser requests to what we serve
blog/2021/03/10-nasa-announcement /blog/2021-03-10-nasa-announcement
blog/2021/05/03-noaa-nwfsc-champions /blog/2021-05-03-noaa-nwfsc-champions
blog/2022/02/17-esip-winter-2022 /blog/2022-02-17-esip-winter-2022

We need to instead create a netlify.toml file, which looks like this:

[[redirects]]
  from = "/blog/2021/03/10-nasa-announcement/" 
  to = "/blog/2021-03-10-nasa-announcement/"
  status = 301

[[redirects]]
  from = "/blog/2021/05/03-noaa-nwfsc-champions/" 
  to = "/blog/2021-05-03-noaa-nwfsc-champions/"
  status = 301
  
[[redirects]]
  from = "/blog/2022/02/17-esip-winter-2022/" 
  to = "/blog/2022-02-17-esip-winter-2022/"
  status = 301

Below are my steps for using regex to help streamline reformatting:

1. In RStudio, open the redirects-list file (where we've stored all redirects for fiddling), then use command + f to open find/replace
2. To capture the first half (i.e. from), Find: blog/([0-9]{4})/([0-9]{2})/([0-9]{2})-([\w-]+) , Replace: from = "blog/\1/\2/\3/\4/"\r (e.g. converts blog/2021/03/10-nasa-announcement to from = "/blog/2021/03/10-nasa-announcement/" and inserts a carriage return)
3. To capture the second half (i.e. to), Find: /blog/([0-9]{4})-([0-9]{2})-([0-9]{2})-([\w-]+), Replace: to = "/blog/\1-\2-\3-\4/"\r (e.g. converts /blog/2021-03-10-nasa-announcement to to = "/blog/2021-03-10-nasa-announcement/" and insets a carriage return. NOTE: Replace All did not insert carriage return, but clicking Replace each time did...unsure why, but ended up just clicking Replace through all occurrences)

Resulting output:

from = "blog/2021/03/10/nasa-announcement/"
to = "/blog/2021-03-10-nasa-announcement/"

from = "blog/2021/05/03/noaa-nwfsc-champions/"
to = "/blog/2021-05-03-noaa-nwfsc-champions/"

from = "blog/2022/02/17/esip-winter-2022/"
to = "/blog/2022-02-17-esip-winter-2022/"

4. To add [[redirects]] above each from/to pair, Find: (from = "/blog/[0-9]{4}/[0-9]{2}/[0-9]{2}/[\w-]+/"), Replace: [[redirects]]\r\1

Resulting output:

[[redirects]]
from = "blog/2021/03/10/nasa-announcement/"
to = "/blog/2021-03-10-nasa-announcement/"

[[redirects]]
from = "blog/2021/05/03/noaa-nwfsc-champions/"
to = "/blog/2021-05-03-noaa-nwfsc-champions/"

[[redirects]]
from = "blog/2022/02/17/esip-winter-2022/"
to = "/blog/2022-02-17-esip-winter-2022/"

5. To add status = 301 after each from/to pair, Find: (to = "/blog/[0-9]{4}-[0-9]{2}-[0-9]{2}-[\w-]+/"), Replace: \1\rstatus = 301

Resulting output:

[[redirects]]
from = "blog/2021/03/10/nasa-announcement/"
to = "/blog/2021-03-10-nasa-announcement/"
status = 301

[[redirects]]
from = "blog/2021/05/03/noaa-nwfsc-champions/"
to = "/blog/2021-05-03-noaa-nwfsc-champions/"
status = 301

[[redirects]]
from = "blog/2022/02/17/esip-winter-2022/"
to = "/blog/2022-02-17-esip-winter-2022/"
status = 301

6. To add tabs ahead of each from/to/status lines, find and replace the following:

• Find: (from = "/blog/[0-9]{4}/[0-9]{2}/[0-9]{2}/[\w-]+/"), Replace: \t\1 (also realized that $ works too, e.g. \t$1)
• Find: (to = "/blog/[0-9]{4}-[0-9]{2}-[0-9]{2}-[\w-]+/"), Replace: \t\1
• Find: (status = 301), Replace: \t\1

Resulting output:

[[redirects]]
  from = "/blog/2021/03/10-nasa-announcement/" 
  to = "/blog/2021-03-10-nasa-announcement/"
  status = 301

[[redirects]]
  from = "/blog/2021/05/03-noaa-nwfsc-champions/" 
  to = "/blog/2021-05-03-noaa-nwfsc-champions/"
  status = 301
  
[[redirects]]
  from = "/blog/2022/02/17-esip-winter-2022/" 
  to = "/blog/2022-02-17-esip-winter-2022/"
  status = 301

[jules32]

Thanks for documenting this all Sam! Let me know if you'd like me to test anything when you merge to main . Edited - Just saw your slack message. I'll build locally first!

[jules32]

I merged #37 to main and did a few quick tests – my first few spot-checks worked, I think we might be set!

Success testing:

• https://www.openscapes.org/blog/2023/01/24/noaa-nmfs-fall/
• https://www.openscapes.org/blog/2022/02/17/esip-winter-2022/

@samanthacsik , I think we're ready for the events! Thank you!

Note for @stefaniebutland as we keep spot-checking/poking around:
The blogs listed in _redirects is convenient since it's a full list, but you have to create the url yourself (i.e. add https://openscapes.org/. I got 404's because of extra //'s.) You can also open each blog post and take the alias from the yaml front matter from the top of the file. For example:
https://github.com/Openscapes/website-new/blob/main/blog/2018-11-20-introducing-openscapes/index.qmd#L11

[jules32]

@samanthacsik when you've done the events and merged to main, will you tag Stef here and she'll help do final spot-checks for broken links. Thank you!