First of all, create your custom error page. You can either use static html or jekyll template for this. If you decide to use one of your custom layout, the path will be a bit different when jekyll generates static pages.

I created a 404.html file that looks like this.

---
layout: default
title: "404: Page not found"
---

Olps.

Now if you create one like mine above at root, it will be generated as 404/index.html so use this path instead of just 404.html when configure nginx.

Edit your nginx config file to add these lines inside server block.

error_page 404 /404/index.html;
error_page 500 502 503 504 /50x/index.html;

Test config your new nginx conf, reload (or restart) and you’re good to go.