LMS - 502 bad gateway invalid request block size: 8518 (max 8192)...skip

[baoc75]

Do Nginx bị lỗi "upstream sent too big header while reading response header from upstream" vì cookie size khá to, với lỗi này, em nghĩ có thể cần tăng proxy buffer size cho courses.steamforvietnam.org

proxy_buffer_size 256k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;

=> Anh @maitrungduc1410 đã sửa nginx config và hết lỗi này

Bây giờ lại thấy lỗi: invalid request block size: 8518 (max 8192)...skip
https://serverfault.com/questions/1056674/invalid-request-block-size-with-nginx-and-uswgi
Cần phải tăng buffer size của web server lên

Anh Đức đã tăng như sau:

extend docker image của LMS hiện tại, update lại CMD, tăng buffer lên.
__
Ok nhất là phải update vào cái docker image mà mình build lúc đầu ấy. Cơ mà a ko biết là mình build image từ Dockerfile nào, cái nào là bản mới nhất @@

a tìm đc CMD ở đây, mặc đunhj buffer là 8192
a update thành 32768

https://github.com/STEAMforVietnam/steam-lmx/blob/main/prod-setup/docker/Dockerfile_koa

mà cái Dockerfile đó cũ quá r, sửa lẫn cuối từ 3 năm trc, ko biết có phải bản chính ko :)))), hỏi chị Lý chị ấy cũng ko nhớ :v

cURL có thể reproduce

curl 'https://courses.steamforvietnam.org/auth/complete/keycloak/?redirect_state=nhuQq31jARfS9pEFv35pQ6OlBJT8WW8O&state=nhuQq31jARfS9pEFv35pQ6OlBJT8WW8O&session_state=907c819d-5896-4267-89da-660e54b21301&code=d878bc33-18fe-4a7d-8cb9-862cddd5d85d.907c819d-5896-4267-89da-660e54b21301.bed41d97-1a7c-4c02-b993-4ea4a01fae49' \
  -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7' \
  -H 'accept-language: vi-VN,vi;q=0.9,fr-FR;q=0.8,fr;q=0.7,en-US;q=0.6,en;q=0.5' \
  -H 'cache-control: max-age=0' \
  -H 'cookie: _ga_9QVQTTBX7B=GS1.1.1686055424.1.1.1686056836.0.0.0; _hjSessionUser_3292725=eyJpZCI6ImY0YTY3NDZhLTQ3MjYtNTMyNC05NTZmLWEyODY2NjZiN2JhYSIsImNyZWF0ZWQiOjE2ODc2ODMwOTYwNTAsImV4aXN0aW5nIjpmYWxzZX0=; steamvn-csrftoken=TjGHjyADRxVJGGiksMAyKyxugf0z0IF2MOSF5K4mF1f0TecNInpENPZp2NYiMsvV; csrftoken=5UHdyXvZcMNJEA0XpnlAAMbEx5j2ziVOtbEJMf58l5PKNiooRyNiytquFvDLQqBJ; steamid-jwt=eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJ6bEFWVHFHbGY3VnlJc0hCX05UelI0NGdYMnBfdUV5SDh3UTA3dmMwcEo4In0.eyJleHAiOjE3MTE1MDk1ODEsImlhdCI6MTcxMTUwNTk4MSwiYXV0aF90aW1lIjoxNzExNTA1OTgxLCJqdGkiOiJiYmNjZTIxYi0yZjRmLTQ2MjktYTI1Zi1lYjBkOGY5MDJhMWMiLCJpc3MiOiJodHRwczovL2lkLnN0ZWFtZm9ydmlldG5hbS5vcmcvcmVhbG1zL1NURUFNIiwiYXVkIjpbInN0ZWFtZXNlIiwib3BlbmVkeCIsInN0ZWFtaWZ5IiwibGl2ZWFwcCIsImFjY291bnQiXSwic3ViIjoiNjg1NmEwYzYtMmI1Zi00Nzc4LWEwZmQtOWRkYWJlNjc4ZWE0IiwidHlwIjoiQmVhcmVyIiwiYXpwIjoic3RlYW1pZnkiLCJub25jZSI6ImY5MjhiYjY4LWZlM2MtNDhmMy04MTkzLWMwMzA3MTBiNDM4ZiIsInNlc3Npb25fc3RhdGUiOiJlMDM1YTQxMy0zMzNlLTQ3YmItYTg0OS1lMzg5N2JiOTIxZTUiLCJhY3IiOiIxIiwiYWxsb3dlZC1vcmlnaW5zIjpbImh0dHBzOi8vc3RlYW1pZnkuc3RlYW1mb3J2aWV0bmFtLm9yZyJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsib2ZmbGluZV9hY2Nlc3MiLCJ1bWFfYXV0aG9yaXphdGlvbiIsImRlZmF1bHQtcm9sZXMtc3RlYW0iXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6Im9wZW5pZCBMaXZlYXBwX0F1ZGllbmNlIGVtYWlsIFN0ZWFtZXNlX0F1ZGllbmNlIHByb2ZpbGUgU3RlYW1pZnlfQXVkaWVuY2UgT3BlbmVkeF9BdWRpZW5jZSIsInNpZCI6ImUwMzVhNDEzLTMzM2UtNDdiYi1hODQ5LWUzODk3YmI5MjFlNSIsImVtYWlsX3ZlcmlmaWVkIjp0cnVlLCJuYW1lIjoiUGhhbiBUaOG7iyBMw6oiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJzNHZfbGVfcGhhbiIsImdpdmVuX25hbWUiOiJQaGFuIFRo4buLIEzDqiIsImVtYWlsIjoicGhhbmxlQHN0ZWFtZm9ydmlldG5hbS5vcmcifQ.MNx0mfvFEJyZwhymzjIZlp9HlYW3mH_LapLADEJ1pZyqhKTn_wCVul7gEdhGqHZiwu-3XUu5lBepJAB8k5cGxXEQDBqCSIPsKRH3a1KjIZcegVv4AI1hHdJSpWghRkz97DHPnS4wjERk0idG8JwCqqEMeXj17WMyLrBMt3YYFFzbLsHreM8MY5Ez6aPblpZGT41597b5Gyjm8ipchBsmzaSi9WQRDChFZFrSQNkIrV4ZEuz9muJg2inXnCz58dGKeRrevBmw5KofYVihLB2XLfQMuOy0dBp0kNpH4oOhhCKbMkbhkN-kdcpE5TPINGQmYeJFBJqVZiZ-Q84Ut9lpMw; _gcl_au=1.1.1277530158.1712919217; experiments_is_enterprise=false; _gid=GA1.2.565050096.1713950679; edxloggedin=true; edx-user-info="{\"version\": 1\054 \"username\": \"S4V_Le_Phan\"\054 \"header_urls\": {\"logout\": \"https://courses.steamforvietnam.org/logout\"\054 \"account_settings\": \"https://courses.steamforvietnam.org/account/settings\"\054 \"learner_profile\": \"https://courses.steamforvietnam.org/u/S4V_Le_Phan\"\054 \"resume_block\": \"https://courses.steamforvietnam.org/courses/course-v1:STEAMforVietnam+SH101+Summer2023/jump_to/block-v1:STEAMforVietnam+SH101+Summer2023+type@html+block@ffe9fc5db22d4f69abc4336103c6a3ef\"}\054 \"user_image_urls\": {\"full\": \"https://courses.steamforvietnam.org/static/images/profiles/default_500.4215dbe8010f.png\"\054 \"large\": \"https://courses.steamforvietnam.org/static/images/profiles/default_120.4a5e0900098e.png\"\054 \"medium\": \"https://courses.steamforvietnam.org/static/images/profiles/default_50.3455a6581573.png\"\054 \"small\": \"https://courses.steamforvietnam.org/static/images/profiles/default_30.deee7287e843.png\"}}"; edx-jwt-cookie-header-payload=eyJhbGciOiJSUzUxMiIsImtpZCI6Im9wZW5lZHgifQ.eyJhdWQiOiAib3BlbmVkeCIsICJleHAiOiAxNzEzOTU0MjkzLCAiaWF0IjogMTcxMzk1MDY5MywgImlzcyI6ICJodHRwczovL2NvdXJzZXMuc3RlYW1mb3J2aWV0bmFtLm9yZy9vYXV0aDIiLCAicHJlZmVycmVkX3VzZXJuYW1lIjogIlM0Vl9MZV9QaGFuIiwgInNjb3BlcyI6IFsidXNlcl9pZCIsICJlbWFpbCIsICJwcm9maWxlIl0sICJ2ZXJzaW9uIjogIjEuMi4wIiwgInN1YiI6ICI4ZmJhMDFkMWFhODZmZmE0ZWNkNmRjYjA5MzgxY2IxNiIsICJmaWx0ZXJzIjogWyJ1c2VyOm1lIl0sICJpc19yZXN0cmljdGVkIjogZmFsc2UsICJlbWFpbF92ZXJpZmllZCI6IHRydWUsICJ1c2VyX2lkIjogMzQyMDgsICJlbWFpbCI6ICJwaGFubGVAc3RlYW1mb3J2aWV0bmFtLm9yZyIsICJuYW1lIjogIlBoYW4gVGhcdTFlY2IgTFx1MDBlYSIsICJmYW1pbHlfbmFtZSI6ICIiLCAiZ2l2ZW5fbmFtZSI6ICJQaGFuIFRoXHUxZWNiIExcdTAwZWEiLCAiYWRtaW5pc3RyYXRvciI6IGZhbHNlLCAic3VwZXJ1c2VyIjogZmFsc2V9; edx-jwt-cookie-signature=U67Xkg8rEUq_4Az3COneOXs3ClaUI-cSfWO7jrELzJMxhKk6oecUm0uX1NBjI0E3O4Jg6m9ih3XCRXm5ofl_e8N853XSs8sC8J1ljndcICvp7vA1VaePS7Y_zkKymaK3IhSx34K1peu-oSM4pghJgf4yTzuLBarmArDZsBN56zoiyKkc26bff56_GvvKl4KDSqAEB9irAvFEhfy526Uo-b7A-_ypeIrpeG9xk9NRK-tVa2QxcfRPBX9NV3wO_wX4Wjn8GbIViL9O1aHpl_V5AxbkcUOPrGVg3-q_UA01gS8c_DusPVXIzQOCtF8ldZ8dmKPb-mbb6ikScS0tMhgz5A; openedx-language-preference=vi; __Secure-next-auth.session-token=eyJhbGciOiJkaXIiLCJlbmMiOiJBMjU2R0NNIn0..1fKLtL4ZTSeDA3q_.3Ufw1ew9Az19B0VijQdSVB-ZVb1Qsvu1QHCrrwhPQExrbHWNYK_-hBLd2RJlKJf66g7l7HFKDfMDvQCzsbHQ4tx-FBtuepCy5ZHV07Lxol7sGZYnP9oUb3-kS2bGE2j8L8CJgfEtcma8LGCMBHB_Gi5_Y95jrDiUFHFkULQ1KUgHwSpQO7xV91sfSvqzFFIsu_SOYei0L4CgNDKqg6bbACugZpWIU2UutD9xll7DsDaZdE0kDRioSzWmGSnJS6YegfQwKGM2QSkhzfatzX7oVBE23WCs_NRpYpjBW-CgdIxNGD9EJMl5n94zmcFQJgjcd4gzD59WVfBLJsuToXMrvbWCLPdBdLYZMhVl9qJM_tLdI0By4-IIbOxYN65yGKFsjCvlk9tU25RWDuosGnDn3q6aUEZIAAQvTZ-IAg52mvQkc2daBaxwbWfOKEDbz_IUaz-psZ2h2KoMH9Ez-a2-3GL8imirRlnVu3PheWT9KWI1R8cU6eOyw_gbaMSV0nragIBHmc722tLMzue9F80lItiZKq_7bv6bWhHEPcRFjWWmkZgHtS6mDFzkkXxpdCrv9j2tR_3o6quhE6NQqvE-snQXUK3iifGcMBi77nERiW6E2ILY0fFYBV4aO5mtKE_PC4b_nC8GpoekJ-28g5r1J1bBULMtzF-Pjhs4mhtFNeYATHludrq7VeL_gehZgQtHe4S5Y6VaAfky-ffMi7Um3jjm9YhE08ZtXQKpkeSM9Z4EVqsRPIJ2PfdX3iYaUDQYYzxQxNOvQt4FNnyZxq6t8Vqr694KUOQ8_eleqjUi5ZIbAYHktUWz3RhuD0XzArATs8roC0XjD7I-g8H_gHx978EM_-sETB_j9qXmfi6nTD_8LqySgmEDe2dozH3F4eE35_V8ARKpfQrb4QykWvrTquZddFjWuI6rTQczpjZ9ltQ1iz5-VV5fNpLPcequciy1v2Sea85tbRFYRUN4zi2mR8dpyx2NTpvAIXidvpkjz7BqUhncAiW_FG0n-MLGV2b4NGdmXb92pc51cBdWzoyw5Qkb39NyQ9k-1vsgU-TUkQdSBbFiQ5aJG08uet1bmYMiRLNdnhizdB4IjDFDH1gfbIGKARMBMJkDAJJ38H6d-C9BHgdLmcAmxlMcBEkUyWEUFNvXImwatPiOnNgAFyrx3s7RuBfd36txbsDQ5Eq2Ugup50UzHZA6rEyTyyBAOCOcgs4KDJX-x5bYaD2mu7VgVARNsjHhjlYWAzb3MK5fz1dGo5DkbCy1gu0_WUOUd51WyvGQMZtlc3UWK8AAzuD5sPcKn8fEaB48FPPcR-moU824tykzECt5TgxMKbTg6I928zNwTOa8reNtau1cQEsEFXkc4kg5HNr2cMQ3yWsbvO8Mrmlj15d8cYanFp-ekLnDHnKSQhzJbs9v-5PpHqSdS_waLFypJzndWubR4V1dc12QcibB35MkqgRNKPoTXBojfweGpBM0rUJa5hHEwJj3BaJHdFpcLIBaicN8Uv0tBWIe3oeZ1DhsxGz7j_wsFCJK8VvDzKyBfTJ2lgy-ZZWA9CiMugVyu7UT9yd3NfxbDvoQaRJ_Ma3XD8RJcVk5tKZXqlIM-YFZxjycXp3Oe-Kw7FHC345GJw2LR7ITrPdnASJUpnapk-IN64nrdNsYzXr7HcQxpUm291Cy18RzPPcKc4euliarmzAjkR4EWdcti94gXMQ6y5un_v2TV2OlMsETjpwcfBJDyfPooad2RMg4wRvOfirAyzF1oyzLzXmukfWLFh0yT2aHbWnZ_Sbc6NIcsqrPp6TrBBNMf1IYN-gj4VQ78SBVUHBaf5jBQnFK5rHh1NyRTBttX5mvI2rgTx9nGlt89Ghzmd2qDlmeof5RRnJ1XNLh7zL9RXXRJw.nhaqt-uZz6soxXvRm4cO2g; _ga_7YMTKLZQG4=GS1.1.1713950562.25.1.1713952719.46.0.0; _ga=GA1.2.1338954608.1681595798; _ga_HF4MG3F3JS=GS1.2.1713950679.7.1.1713952727.0.0.0; sessionid=1|n1t55bti1z0ui38bggxb9llfb7sonuuk|UUW1XGVCLfn2|IjVjNThiNTcyYjE0YTM5YWZiNGVmYzBjYzBjMjI1MzAyMGMxNzMyM2ExNDY3ZTBkMWIyYjNlZTUzZjc1N2U0MmUi:1rzZOw:8VLB66Ely-pUCR7dmhD6vKW7fAw' \
  -H 'priority: u=0, i' \
  -H 'sec-ch-ua: "Chromium";v="124", "Google Chrome";v="124", "Not-A.Brand";v="99"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Windows"' \
  -H 'sec-fetch-dest: document' \
  -H 'sec-fetch-mode: navigate' \
  -H 'sec-fetch-site: same-origin' \
  -H 'sec-fetch-user: ?1' \
  -H 'upgrade-insecure-requests: 1' \
  -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36'

[maitrungduc1410]

Lí do: invalid request block size: 8518 -> server uwsgi chạy LMS đang start với option --buffer-size=8192 -> request gửi tới lớn hơn buffer size -> close connection

Tham khảo CMD start uwsgi ở đây

Có thể kiểm chứng chính xác bằng cách exec vào contaner trên k8s đang chạy lms (namespace lilac, deployment openedx-lms):

ps aux | grep uwsgi

>>>>
root         122  0.2  0.8 1410268 266748 ?      S    03:11   0:00 uwsgi --static-map /static=/openedx/staticfiles/ --static-map /media=/openedx/media/ --http 0.0.0.0:8000 --thunder-lock --single-interpreter --enable-threads --processes=2 --buffer-size= 8192 --wsgi-file lms/wsgi.py
root         130  0.0  0.0   9044   724 pts/0    S+   03:16   0:00 grep uwsgi

[maitrungduc1410]

Cách fix: update lại buffer size limit khi start uwsgi

Cách 1: update trực tiếp deployment của lms trên k8s (đơn giản nhất)

containers:
  - name: openedx-lms
    image: registry.digitalocean.com/steamvn/openedx:lilac_1.3
    command:
      - uwsgi
    args:
      - "--static-map"
      - /static=/openedx/staticfiles/
      - "--static-map"
      - "/media=/openedx/media/"
      - "--http"
      - "0.0.0.0:8001"
      - "--thunder-lock"
      - "--single-interpreter"
      - "--enable-threads"
      - "--processes=2"
      - "--buffer-size=32768"
      - "--wsgi-file"
      - "lms/wsgi.py"

Ở trên đã dùng chính xác CMD như image mặc định, start lên vào kiểm chứng bằng ps aux | grep uwsgi cho kết quả y hệt, nhưng liên tục gặp lỗi module not found lms.envs.aws khi load file lms/wsgi.py)

Dồn hết vào command hay tách ra command và args đều không được

Cách 2: extend image FROM image đang chạy ở production, override CMD -> build -> push

FROM --platform=linux/amd64 registry.digitalocean.com/steamvn/openedx:lilac_1.3
CMD uwsgi \
    --static-map /static=/openedx/staticfiles/ \
    --static-map /media=/openedx/media/ \
    --http 0.0.0.0:8000 \
    --thunder-lock \
    --single-interpreter \
    --enable-threads \
    --processes=${UWSGI_WORKERS:-2} \
    --buffer-size=32768 \
    --wsgi-file ${SERVICE_VARIANT}/wsgi.py

--platform=linux/amd64: nếu build image trên Macbook Apple Chip

Note

Hiện tại đang dùng Cách 2, image build local và push thẳng lên DO, tên image đang dùng: registry.digitalocean.com/steamvn/openedx:lilac_1.3_increase_buffer

Khá khó hiểu vì cách update trực tiếp vào manifest K8S lại không được dù đã thử nhiều hướng khác nhau, về mặt kĩ thuật nó cũng không khác với việc extend Docker image

[maitrungduc1410]

Updated K8S manifests: https://github.com/STEAMforVietnam/lms-deploy/commit/98ad519e4da2ddb392fd920fa9658ecbe5ed99f3