Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

The custom plug-in returns the result garbled #1559

Closed
griffenliu opened this issue May 9, 2020 · 10 comments · Fixed by #1598
Closed

The custom plug-in returns the result garbled #1559

griffenliu opened this issue May 9, 2020 · 10 comments · Fixed by #1598
Assignees
Labels
good first issue Good for newcomers

Comments

@griffenliu
Copy link
Contributor

code:

function _M.access(conf, ctx)
    ...
    log.warn(">>>>>>>>>>>>>>>>>>", data)
    local accept_type = ngx.req.get_headers()["Accept"]
    if str_contains(accept_type, "text/html") then
        return 200, data
    else
        return 200, decode(data)
    end
end

The data comes from the cache

log:

2020/05/09 10:55:59 [warn] 23640#0: *812 [lua] dfy-cache-plugin.lua:380: phase_fun(): >>>>>>>>>>>>>>>>>>{"data":"测试服务器8001"}
, client: 172.30.212.72, server: , request: "GET /read?_YCD_={%22read%22:%20[%2211111111111%22]} HTTP/1.1", host: "172.30.212.72:9080"

result:

{"data":"测试�务器8001"}

I think the reason is that the encoding is not configured in nginx.conf. Can we add utf-8 encoding into it or specify the encoding through configuration?

@moonming
Copy link
Member

I think the reason is that the encoding is not configured in nginx.conf.

can you show a example? I don't konw how to add utf-8 encoding in nginx.conf.

@griffenliu
Copy link
Contributor Author

http {
include mime.types;
default_type text/html;
charset utf8;
}

@moonming
Copy link
Member

moonming commented May 11, 2020 via email

@membphis membphis added beginner good first issue Good for newcomers labels May 12, 2020
@xxm404
Copy link
Contributor

xxm404 commented May 12, 2020

Can I modify APISIX Nginx template to solve this problem

@moonming
Copy link
Member

moonming commented May 12, 2020 via email

@xxm404
Copy link
Contributor

xxm404 commented May 12, 2020

ok please assign this issue to me.

@xxm404
Copy link
Contributor

xxm404 commented May 13, 2020

Add charset utf-8; to APISIX Nginx template
image
Modify example-plugin to test
image
restart apisix and execute curl -i 127.0.0.1:9080/api/bench/testSimple It still garbled
image

@xxm404
Copy link
Contributor

xxm404 commented May 14, 2020

Hi @griffenliu can you give me a minimal reproducible example, I can not reproduce this problem
In my test, the response normally

my test code

function _M.access(conf, ctx)
    -- core.log.warn("plugin access phase, conf : ", core.json.encode(conf))
    -- return 200, core.json.decode(data)
    local redis = redis_new()
    local ok, err = redis:connect('127.0.0.1', 6379)
    if not ok then
        return false, err
    end
    local data = redis:get('data')
    core.log.warn(">>>>>>>>>>>>>>>", data)
    return 200, core.json.decode(data)
end

log

2020/05/14 13:02:55 [warn] 17681#17681: *332 [lua] example-plugin.lua:70: phase_fun(): >>>>>>>>>>>>>>>{"data":"测试服务器8001"}, client: 127.0.0.1, server: , request: "GET /api/bench/testSimple HTTP/1.1", host: "127.0.0.1:9080"

response

HTTP/1.1 200 OK
Date: Thu, 14 May 2020 13:04:23 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: keep-alive
Server: APISIX web server

{"data":"测试服务器8001"}

@griffenliu
Copy link
Contributor Author

sorry, I'm late. There is no problem with your example, which is the same as mine. The encoding was normal when I used curl to test, but it was garbled when I used the browser.

Using the browser response results below, you can see the difference between a correct response and an error response:

# this is garbled response
Connection: keep-alive
Content-Type: text/plain
Date: Mon, 18 May 2020 09:53:34 GMT
Server: APISIX web server
Transfer-Encoding: chunked
# this is normal response,  Content-Type with encoding: charset=utf8.
Connection: keep-alive
Content-Type: text/plain; charset=utf8
Date: Mon, 18 May 2020 09:55:06 GMT
Server: APISIX web server
Transfer-Encoding: chunked

@griffenliu
Copy link
Contributor Author

Hi @griffenliu can you give me a minimal reproducible example, I can not reproduce this problem
In my test, the response normally

my test code

function _M.access(conf, ctx)
    -- core.log.warn("plugin access phase, conf : ", core.json.encode(conf))
    -- return 200, core.json.decode(data)
    local redis = redis_new()
    local ok, err = redis:connect('127.0.0.1', 6379)
    if not ok then
        return false, err
    end
    local data = redis:get('data')
    core.log.warn(">>>>>>>>>>>>>>>", data)
    return 200, core.json.decode(data)
end

log

2020/05/14 13:02:55 [warn] 17681#17681: *332 [lua] example-plugin.lua:70: phase_fun(): >>>>>>>>>>>>>>>{"data":"测试服务器8001"}, client: 127.0.0.1, server: , request: "GET /api/bench/testSimple HTTP/1.1", host: "127.0.0.1:9080"

response

HTTP/1.1 200 OK
Date: Thu, 14 May 2020 13:04:23 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: keep-alive
Server: APISIX web server

{"data":"测试服务器8001"}

Thanks for helping me solve my problem :)

SaberMaster pushed a commit to SaberMaster/incubator-apisix that referenced this issue Jun 30, 2020
nic-chen added a commit to nic-chen/apisix that referenced this issue Jul 6, 2020
* feature: implemented plugin `sys logger`. (apache#1414)

* bugfix(CORS): using rewrite phase and add lru cache for multiple origin  (apache#1531)

* change: updated the dashboard submodule to latest version. (apache#1540)

* doc: alter logger plugins documentations. (apache#1541)

* bugfix: Adding function to remove stale objects from kafka logger (apache#1526)

* bugfix: removed stale objects from tcp logger (apache#1543)

* bugfix: removing stale objects from udp logger (apache#1544)

* optimize: use buffer for plugin `syslog`. (apache#1551)

* plugin: add HTTP logger for APISIX  (apache#1396)

* bugfix: got 500 error when using post method in grpc-transcode plugin(apache#1566)

* bugfix: removed stale object in sys log. (apache#1557)

* feature(prometheus): support to collect metric `overhead` (apache#1576)

Fix apache#1534 .

* feature: support new field `exptime` for SSL object. (apache#1575)

fix apache#1571.

* doc: Added FAQ about how to reload your own plugin (apache#1568)

* doc: repair the white paper's url of README (apache#1582)

* chore: fix function name typo in ip-restriction (apache#1586)

* doc: added http logger Chinese docs (apache#1581)

* feature: support discovery center (apache#1440)

* doc:add chinese version for install doc (apache#1590)

* bugfix: incorrect variable name `hostCount` (apache#1585)

* doc: update kakfa logger plugin's cn version (apache#1594)

* doc: fix the doc style for *_logger.md (apache#1605)

* bugfix: raise error when none of the configured etcd can be connected (apache#1608)

Close apache#1561.

* test: updated style. (apache#1606)

* release: released 1.3 version. (apache#1558)

* bugfix(CLI): fixed garbled Chinese response in browser.  (apache#1598)

fix apache#1559

* change: updated prometheus to version 1.1 . (apache#1607)

* doc: add asf.yaml. (apache#1612)

* fix some doc style for response-rewrite* and health-check.md (apache#1611)

* makefile: add default check for install command (apache#1591)

* test cases: add doc and test cases for how to redirect http to https. (apache#1595)

* add FAQ about redirect http To https

* add test cases for serverless plugin and redirect plugin

Co-authored-by: rhubard <18734141014@163.com>

* feature: add skywalking plugin. (apache#1241)

* doc: removed external links and docs. (apache#1619)

* doc: add coc file (apache#1589)

* bugfix: change the version of skywalking to 1.0-0 (apache#1624)

* bugfix(prometheus): the `overhead` should use milliseconds. apache#1615 (apache#1616)

Fix apache#1615

* feature: add option to include request body in log util (apache#1545)

* bugfix: fix typo of `instance_id` in skywalking plugin. (apache#1629)

* doc: added the link to discovery.md (apache#1631)

* change(ASF): add notifications to mailing list. (apache#1635)

* change(doc): style for HttpResponse section (apache#1634)

* doc(limit-count): fixed document description does not match source code.  (apache#1628)

close apache#1627

* bugfix(batch-requests): support cookie (apache#1599)

* feat(admin api): enhance `PATCH` method, allow to update partial data. (apache#1609)

* test: added test tests for skywalking. (apache#1621)

* doc: add skywalking plugin instructions (apache#1636)

* feature: support http_to_https in redirect plugin. (apache#1642)

* test: add test case for apache#1625 to test the filed of overhead (apache#1645)

* CLI: compatibility of benchmark script and apisix reload command on OSX (apache#1650)

* feature:  support to enable HTTPS for admin API (apache#1648)

* [log] Optimize the buffer size and flush time (apache#1570)

* yousali:<log>Optimize the buffer size and flush time

1. buffer=4096 is better for Writes of more than PIPE_BUF bytes may be nonatomic
2. flush=1. Since the log buffer is lowered, the flush time should also be lowered.

* yousali:<fix>

hi, I also made a test.

```
4096  Requests/sec:  16079.75
8192 Requests/sec:  16389.52
16384 Requests/sec:  16395.30
32768 Requests/sec:  16459.71
```
I think a log buffer size of 8192 or 16384 would be appropriate.

On the other hand, the refresh time of 3 seconds is still relatively long, and 1 or 3 seconds doesn't particularly affect QPS.

So I also agree with `buffer=16384 flush=1; `

* doc: add 'X-API-KEY' parameter for each interface of Admin API. (apache#1661)

* bugfix: wildcard certificates cannot match multi-level subdomains in … (apache#810)

* plugin: add consumer-restriction (apache#1437)

* feat: support resource name for route, service and upstream object. (apache#1655)

* [bugfix(CLI)]: check whether  the user has enabled etcd v2 protocol. (apache#1665)

* bugfix(CLI): generate the 'worker_cpu_affinity' config for Linux OS (apache#1658)

Fix apache#1657

* test case: formatted by `reindex`. (apache#1651)

* change: disable reuseport in development mode, it more easy to manage worker process. (apache#1175)

* test: add test case for route with `filter_func`. (apache#1683)

* doc: rename grpc-transcoding-cn.md to grpc-transcode-cn.md (apache#1694)

* fix bug: Execute command 'make run' multiple times, will start multiple processes (apache#1692)

Fix apache#1690

* doc(FAQ): added example for gray release. (apache#1687)

* change: set default reject code for some plugins (apache#1696)

plugin list:

limit-count
limit-conn
limit-req

* feature: ssl enhance (apache#1678)

support enable or disable ssl by patch method
support encrypted storage of the SSL private key in etcd
support multi snis

Fix apache#1668

* feature:  support body filter plugin `echo`. (apache#1632)

* doc: Update README_CN.md (apache#1705)

* change: use `iterate` to scan items in etcd.  (apache#1717)

related issue: apache#1685

* doc: added doc of key for limit-* plugins. (apache#1714)

* feature: support authorization Plugin for Keycloak Identity Server (apache#1701)

* feat[batch-request]: cp all header to every request (apache#1697)

* doc: updated main picture. (apache#1719)

* doc: update echo-cn.md (apache#1726)

* update `resty-etcd` to version 1.0 . (apache#1725)

* doc: health-check-cn.md (apache#1723)

* doc: add Chinese translation of authz-keycloak plugin (apache#1729)

* doc: Refactoring docs to support docsify (apache#1724)

* change: update `resty-radixtree` to version 1.9 . (apache#1730)

* feature: support the use of independent files to implement the load a… (apache#1732)

* feature: support the use of independent files to implement the load algorithm,
which is convenient for expanding different algorithms in the future.

* feature(echo): support header filter and access phases. (apache#1708)

* bugfix: id can be string object, which contains `^[a-zA-Z0-9-_]+$`. (apache#1739)

Fix apache#1654

* test: add test cases about the string id in `service` apache#1659 (apache#1750)

* update `lua-resty-raditree` to ver 2.0 . (apache#1748)

* refactory: collect `upstream` logic and put them in a single file. (apache#1734)

feature: support dynamic upstream in plugin.

here is a mini  example in `access` phase of plugin:

```lua
    local up_conf = {
        type = "roundrobin",
        nodes = {
            {host = conf.upstream.ip, port = conf.upstream.port, weight = 1},
        }
    }

    local ok, err = upstream.check_schema(up_conf)
    if not ok then
        return 500, err
    end

    local matched_route = ctx.matched_route
    upstream.set(ctx, up_conf.type .. "#route_" .. matched_route.value.id,
                 ctx.conf_version, up_conf, matched_route)
    return
```

* feature: implemented plugin `uri-blocklist` . (apache#1727)

first step: apache#1617

* doc: update `http-logger` plugins Chinese docs. (apache#1755)

* doc: update admin-api docs (apache#1753)

* doc: add oauth plugins Chinese docs. (apache#1754)

* bugfix: fixed configures of nginx.conf for security reasons (apache#1759)

removed working_directory and removed TLSv1 TLSv1.1 from ssl_protocols

* doc: update Chinese README.md (apache#1758)

* test: use longer ttl, avoid the cached item expired. (apache#1760)

* doc: updated k8s doc (apache#1757)

* bugfix: Fix for remote open ID connect introspection (apache#1743)

fix apache#1741

* test: added test cases. (apache#1752)

* bugfix: added `content-type` for admin API responses (apache#1746)

* feature: support etcd auth (apache#1769)

Fix apache#1713 , apache#1770

* plugin(heartbeat): use `info` log level when failed to report heartbeat. (apache#1771)

* optimize: Use lru to avoid resolving IP addresses repeatedly . (apache#1772)

* optimize: Use lru to avoid resolving IP addresses repeatedly .
Cached the global rules to `ctx` .

* optimzie: used a longer time interval for etcd and flush access log.

* optimize: return upstream node directly if the count is 1 .

* optimize: avoid to cache useless variable.

* doc: update Chinese README.md (apache#1763)

* doc: remove router `r3` . (apache#1764)

* release: released 1.4-0 version (apache#1742)

* bugfix(config etcd): when we reset the fetched data, `sync_times` also needs to be reset. (apache#1785)

* change: remove authentication type for cors plugin (apache#1788)

fix apache#1787

* rocks: fixed wrong source of 1.4. (apache#1783)

* change: 'get_plugin_list' API sorts the return list base on priority (apache#1779)

* test: format by tool `reindex`. (apache#1775)

* bugfix: missing argument `premature` because it was called by ngx.timer . (apache#1796)

* bugfix: return `404 Not Found` when the dashboard folder is empty.  (apache#1799)

close apache#1794

* doc: add guides for installing dependencies on fedora (apache#1800)

* doc: fixed some punctuation error in the document sample shell (apache#1803)

Co-authored-by: Ayeshmantha Perera <akayeshmantha@apache.org>
Co-authored-by: Vinci Xu <277040271@qq.com>
Co-authored-by: Nirojan Selvanathan <sshniro@gmail.com>
Co-authored-by: YuanSheng Wang <membphis@gmail.com>
Co-authored-by: Yousa <snowfly1993@gmail.com>
Co-authored-by: hiproz <hiproz@gmail.com>
Co-authored-by: 罗泽轩 <spacewanderlzx@gmail.com>
Co-authored-by: Scaat Feng <scaat.feng@gmail.com>
Co-authored-by: qiujiayu <153163285@qq.com>
Co-authored-by: dengliming <liming.d.pro@gmail.com>
Co-authored-by: dabue <53054094+dabue@users.noreply.github.com>
Co-authored-by: Wen Ming <moonbingbing@gmail.com>
Co-authored-by: xxm404 <46340314+xxm404@users.noreply.github.com>
Co-authored-by: rhubard <18734141014@163.com>
Co-authored-by: Gerrard-YNWA <gyc_ssdut@163.com>
Co-authored-by: 月夜枫 <cyxinda@163.com>
Co-authored-by: 仇柯人 <qiuker521@163.com>
Co-authored-by: stone4774 <25053818+stone4774@users.noreply.github.com>
Co-authored-by: 琚致远 <juzhiyuan@apache.org>
Co-authored-by: Kev.Hu <kevhoo@126.com>
Co-authored-by: QuakeWang <45645138+QuakeWang@users.noreply.github.com>
Co-authored-by: agile6v <agile6v@gmail.com>
Co-authored-by: Corey.Wang <wangchao523@126.com>
Co-authored-by: hellmage <luyunxie14@gmail.com>
Co-authored-by: Eric Shi <shibingli@yeah.net>
Co-authored-by: Shenal Silva <shenal@users.noreply.github.com>
Co-authored-by: jackstraw <932698529@qq.com>
Co-authored-by: morrme <morrme@users.noreply.github.com>
Co-authored-by: ko han <hanke0@outlook.com>
Co-authored-by: Joey <majunjiev@gmail.com>
Co-authored-by: YuanYingdong <1975643103@qq.com>
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
good first issue Good for newcomers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants