[Solved] Kobo sync and reverse proxy

[norangebit]

Environment:

• Calibre-Web 0.6.11 with linuxserver docker image.
• The calibre-web conteiner exposes the port 8083.
• Nginx Porxy Manager (NPM) as reverse proxy (exposes port 80 and 443).
• NPM redirect the traffic from books.mydomain to port 8083 of the container
• Kobo Clara HD 4.26.16704.

Describe the bug/problem

The sync between calibre-web and Kobo doesn't always work.
I have run a couple of tests to try to find the problem and I think it has something to do with the reverse proxy.
In fact, I was only able to sync correctly when I used the server's IP directly as the endpoint.

It is possible that the error can be traced back to some configuration error in the proxy server. But for viewing the web app there is no problem, and this user seems to have a similar problem to mine.
I preferred to open a new issue because the OP of the other issue did not use a reverse proxy and had problems all the time, in my case if I use the ip the sync works.

Below are the conditions under which I performed the tests and the calibre logs.

Note: In all tests, NPM was set to force SSL and block exploitation. I have performed further tests, not reported, and it seems that these configurations are not relevant.

Test 1

• Kobo state:
  • No update to sync
• Kobo sync settings:
  • Proxy unknown request enable
  • Port forwarded 8083
  • Endpoint books.<domain>/<token>

Sync result: ERROR

[2021-03-09 12:28:11,589] DEBUG {cps.kobo:830} Cover request received for book d70aafa8-fc94-42d8-9f2e-0bb943fd3c84
[2021-03-09 12:28:12,081]  INFO {cps.kobo:138} Kobo library sync request received.
[2021-03-09 12:28:12,081] DEBUG {cps.kobo:140} Kobo: Received unproxied request, changed request port to external server port
[2021-03-09 12:28:17,365]  INFO {cps.kobo:941} Init
[2021-03-09 12:28:17,487] DEBUG {cps.kobo:956} Kobo: Received unproxied request, changed request port to external server port
[2021-03-09 12:28:17,487] DEBUG {cps.kobo:966} Kobo: Received unproxied request, changed request url to http://books.<mydomain>:8083
[2021-03-09 12:28:17,641] DEBUG {cps.kobo:878} Unimplemented User Request received: http://books.<maydomain>/kobo/<token>/v1/user/profile
[2021-03-09 12:28:18,207] DEBUG {cps.kobo:878} Unimplemented User Request received: http://books.<mydomain>/kobo/<token>/v1/user/loyalty/benefits
[2021-03-09 12:28:18,562] DEBUG {cps.kobo:878} Unimplemented User Request received: http://books.<mydomain>/kobo/<token>/v1/analytics/gettests

Test 2

• Kobo state:
  • No update to sync
• Kobo sync settings:
  • Proxy unknown request disable
  • Port forwarded 8083
  • Endpoint books.<domain>/<token>

Sync result: OK

[2021-03-09 12:34:50,975]  INFO {cps.kobo:941} Init
[2021-03-09 12:34:50,975] DEBUG {cps.kobo:956} Kobo: Received unproxied request, changed request port to external server port
[2021-03-09 12:34:50,976] DEBUG {cps.kobo:966} Kobo: Received unproxied request, changed request url to http://books.<mydomain>:8083
[2021-03-09 12:34:51,127] DEBUG {cps.kobo:878} Unimplemented User Request received: http://books.<mydomain>/kobo/<token>/v1/user/profile
[2021-03-09 12:34:51,269] DEBUG {cps.kobo:878} Unimplemented User Request received: http://books.<mydomain>/kobo/<token>/v1/user/loyalty/benefits
[2021-03-09 12:34:51,399] DEBUG {cps.kobo:878} Unimplemented User Request received: http://books.<mydomain>/kobo/<token>/v1/analytics/gettests
[2021-03-09 12:34:51,700]  INFO {cps.kobo:138} Kobo library sync request received.
[2021-03-09 12:34:51,701] DEBUG {cps.kobo:140} Kobo: Received unproxied request, changed request port to external server port
[2021-03-09 12:34:52,165] DEBUG {cps.kobo:878} Unimplemented User Request received: http://books.<mydomain>/kobo/<token>/v1/user/wishlist
[2021-03-09 12:34:53,017] DEBUG {cps.kobo:892} Unimplemented Products Request received: http://books.<mydomain>/kobo/<token>/v1/products/books/series/46055a14-4f71-3c7b-9c4e-081eb83d47b3
[2021-03-09 12:34:53,280] DEBUG {cps.kobo:878} Unimplemented User Request received: http://books.<mydomain>/kobo/<token>/v1/user/recommendations
[2021-03-09 12:34:54,334] DEBUG {cps.kobo:892} Unimplemented Products Request received: http://books.<mydomain>/kobo/<token>/v1/products/ec4460eb-de31-41ee-b24f-d41e5c88e6ae/nextread
[2021-03-09 12:34:54,468] DEBUG {cps.kobo:892} Unimplemented Products Request received: http://books.<mydomain>/kobo/<token>/v1/products/6b3fe26b-14fd-471b-80e5-df7065546a9f/nextread
[2021-03-09 12:34:54,601] DEBUG {cps.kobo:892} Unimplemented Products Request received: http://books.<mydomain>/kobo/<token>/v1/products/d70aafa8-fc94-42d8-9f2e-0bb943fd3c84/nextread

Test 3

• Kobo state:
  • New book to sync
• Kobo sync settings:
  • Proxy unknown request disable
  • Port forwarded 8083
  • Endpoint books.<domain>/<token>

Sync result: ERROR

[2021-03-09 12:38:59,485]  INFO {cps.kobo:941} Init
[2021-03-09 12:38:59,485] DEBUG {cps.kobo:956} Kobo: Received unproxied request, changed request port to external server port
[2021-03-09 12:38:59,486] DEBUG {cps.kobo:966} Kobo: Received unproxied request, changed request url to http://books.<mydomain>:8083
[2021-03-09 12:38:59,631] DEBUG {cps.kobo:878} Unimplemented User Request received: http://books.<mydomain>/kobo/<token>/v1/user/profile
[2021-03-09 12:38:59,784] DEBUG {cps.kobo:878} Unimplemented User Request received: http://books.<mydomain>/kobo/<token>/v1/user/loyalty/benefits
[2021-03-09 12:38:59,931] DEBUG {cps.kobo:878} Unimplemented User Request received: http://books.<mydomain>/kobo/<token>/v1/analytics/gettests
[2021-03-09 12:39:00,277]  INFO {cps.kobo:138} Kobo library sync request received.
[2021-03-09 12:39:00,278] DEBUG {cps.kobo:140} Kobo: Received unproxied request, changed request port to external server port
[2021-03-09 12:39:00,452]  INFO {cps.tasks.convert:102} Book id 177 - target format of .kepub does not exist. Moving forward with convert.
[2021-03-09 12:39:00,716] DEBUG {cps.tasks.convert:160} Kepubify v3.1.4
[2021-03-09 12:39:00,717] DEBUG {cps.tasks.convert:160} [  1/  1] Converting /books/path/of/book/to/convert.epub
[2021-03-09 12:39:00,717] DEBUG {cps.tasks.convert:160} 1 total: 1 converted, 0 copied, 0 skipped, 0 errored

Test 4

• Kobo state:
  • New book to sync
• Kobo sync settings:
  • Proxy unknown request disable
  • Port forwarded 80
  • Endpoint books.<domain>/<token>

Sync result: ERROR

[2021-03-09 12:43:45,735]  INFO {cps.kobo:941} Init
[2021-03-09 12:43:45,735] DEBUG {cps.kobo:956} Kobo: Received unproxied request, changed request port to external server port
[2021-03-09 12:43:45,735] DEBUG {cps.kobo:966} Kobo: Received unproxied request, changed request url to http://books.<mydomain>:80
[2021-03-09 12:43:45,921] DEBUG {cps.kobo:878} Unimplemented User Request received: http://books.<mydomain>/kobo/<token>/v1/user/profile
[2021-03-09 12:43:46,089] DEBUG {cps.kobo:878} Unimplemented User Request received: http://books.<mydomain>/kobo/<token>/v1/user/loyalty/benefits
[2021-03-09 12:43:46,224] DEBUG {cps.kobo:878} Unimplemented User Request received: http://books.<mydomain>/kobo/<token>/v1/analytics/gettests
[2021-03-09 12:43:46,579]  INFO {cps.kobo:138} Kobo library sync request received.
[2021-03-09 12:43:46,579] DEBUG {cps.kobo:140} Kobo: Received unproxied request, changed request port to external server port

Test 5

• Kobo state:
  • New book to sync
• Kobo sync settings:
  • Proxy unknown request enable
  • Port forwarded 8083
  • Endpoint <ip>:8083/<token>

Sync result: OK

[2021-03-09 12:47:05,760] DEBUG {cps.kobo:830} Cover request received for book d70aafa8-fc94-42d8-9f2e-0bb943fd3c84
[2021-03-09 12:47:12,810] DEBUG {cps.helper:695} unrar version 5.61 beta 1
[2021-03-09 12:47:12,816] DEBUG {cps.config_sql:323} _ConfigSQL updating storage
[2021-03-09 12:47:27,767]  INFO {cps.kobo:941} Init
[2021-03-09 12:47:27,922] DEBUG {cps.kobo:956} Kobo: Received unproxied request, changed request port to external server port
[2021-03-09 12:47:27,922] DEBUG {cps.kobo:966} Kobo: Received unproxied request, changed request url to http://<ip>:8083
[2021-03-09 12:47:28,113] DEBUG {cps.kobo:878} Unimplemented User Request received: http://<ip>/kobo/<token>/v1/user/profile
[2021-03-09 12:47:28,694] DEBUG {cps.kobo:878} Unimplemented User Request received: http://<ip>/kobo/<token>/v1/user/loyalty/benefits
[2021-03-09 12:47:29,067] DEBUG {cps.kobo:878} Unimplemented User Request received: http://<ip>/kobo/<token>/v1/analytics/gettests
[2021-03-09 12:47:29,794]  INFO {cps.kobo:138} Kobo library sync request received.
[2021-03-09 12:47:29,794] DEBUG {cps.kobo:140} Kobo: Received unproxied request, changed request port to external server port
[2021-03-09 12:47:30,310]  INFO {cps.kobo:138} Kobo library sync request received.
[2021-03-09 12:47:30,310] DEBUG {cps.kobo:140} Kobo: Received unproxied request, changed request port to external server port
[2021-03-09 12:47:30,615] DEBUG {cps.kobo:830} Cover request received for book 86c946bf-7fe4-46eb-bb2c-7540b0c0dcc7
[2021-03-09 12:47:30,765] DEBUG {cps.kobo:830} Cover request received for book 86c946bf-7fe4-46eb-bb2c-7540b0c0dcc7
[2021-03-09 12:47:31,555] DEBUG {cps.kobo:282} Kobo: Received unproxied request, changed request port to external server port
[2021-03-09 12:47:31,555]  INFO {cps.kobo:283} Kobo library metadata request received for book 86c946bf-7fe4-46eb-bb2c-7540b0c0dcc7
[2021-03-09 12:47:33,235] DEBUG {cps.kobo:830} Cover request received for book d70aafa8-fc94-42d8-9f2e-0bb943fd3c84
[2021-03-09 12:47:34,081] DEBUG {cps.kobo:878} Unimplemented User Request received: http://<ip>/kobo/<token>/v1/user/wishlist
[2021-03-09 12:47:35,945] DEBUG {cps.kobo:892} Unimplemented Products Request received: http://<ip>/kobo/<token>/v1/products/books/series/46055a14-4f71-3c7b-9c4e-081eb83d47b3
[2021-03-09 12:47:36,470] DEBUG {cps.kobo:878} Unimplemented User Request received: http://<ip>/kobo/<token>/v1/user/recommendations
[2021-03-09 12:47:37,588] DEBUG {cps.kobo:892} Unimplemented Products Request received: http://<ip>/kobo/<token>/v1/products/ec4460eb-de31-41ee-b24f-d41e5c88e6ae/nextread
[2021-03-09 12:47:38,000] DEBUG {cps.kobo:892} Unimplemented Products Request received: http://<ip>/kobo/<token>/v1/products/6b3fe26b-14fd-471b-80e5-df7065546a9f/nextread
[2021-03-09 12:47:38,542] DEBUG {cps.kobo:892} Unimplemented Products Request received: http://<ip>/kobo/<token>/v1/products/d70aafa8-fc94-42d8-9f2e-0bb943fd3c84/nextread

[norangebit]

OK I think I have found the right configuration.
The problem depends in part on kobo and in part on Nginx Proxy Manager.

• Kobo override HTTPS with HTPP on image_host (see Kobo sync error on Docker using https reverse proxy #1758).
• Nginx Proxy Manager uses a incompatible buffer size.

I have listed my configuration below so that it can be of help to other users.

Nginx Proxy Manager

1. Create a new proxy host for calibre-web.
   You can enable force SSL, HSTS and Block Common Exploits without any problems.
2. Go to advanced tab and enter the following parameters:

   proxy_buffer_size 128k;
   proxy_buffers 4 256k;
   proxy_busy_buffers_size 256k;

Calibre-web

1. Go to admin setting and enable Kobo sync.
2. Set Server External Port to 80.
   If the port is set to 443, it is not possible to fetch covers.
3. Go to your profile page, enable Kobo sync and copy the api endpoint.

Kobo

1. Open .kobo/Kobo eReader.conf file in a text editor end edit your api endpoint.
   Make sure you use HTTPS and not HTTP.
   This is an example of a correct line: api_endpoint=https://books.<domain>/kobo/<token>

[dialanothernumb]

I use NPM set up also and tried your settings under the advanced tab. Everything synchronises well. However when I add my oauth server settings to the advanced tab, sync starts but nothing happens.
Here is what I have under the advanced tab. Is there something in then layout or something in the logic that I haven't seen?:

proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;

OAuth Snippet Start ----------------------------------------------------

location /oauth2/ {
proxy_pass http://$server:4180;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Scheme $scheme;
proxy_set_header X-Auth-Request-Redirect $request_uri;
}

location = /oauth2/auth {
proxy_pass http://$server:4180;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Scheme $scheme;
# nginx auth_request includes headers but not body
proxy_set_header Content-Length "";
proxy_pass_request_body off;
}

location / {
auth_request /oauth2/auth;
error_page 401 = /oauth2/sign_in;
# pass information via X-User and X-Email headers to backend,
# requires running with --set-xauthrequest flag
auth_request_set $user $upstream_http_x_auth_request_user;
auth_request_set $email $upstream_http_x_auth_request_email;
proxy_set_header X-User $user;
proxy_set_header X-Email $email;

# if you enabled --cookie-refresh, this is needed for it to work with auth_request
auth_request_set $auth_cookie $upstream_http_set_cookie;
add_header Set-Cookie $auth_cookie;

# Proxy!
include conf.d/include/proxy.conf;

}

Oauth Snippet End

[simkin]

@norangebit did you experience issues with getting redirected the login screen? I am new to NPM as proxy manager and experience this issue (login works, every other page I load redirects to login)