Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

elasticsearch on swarm cluster, ulimits ignored #88

Closed
2 of 3 tasks
pere3 opened this issue Aug 29, 2017 · 2 comments · Fixed by docker/cli#482
Closed
2 of 3 tasks

elasticsearch on swarm cluster, ulimits ignored #88

pere3 opened this issue Aug 29, 2017 · 2 comments · Fixed by docker/cli#482

Comments

@pere3
Copy link

pere3 commented Aug 29, 2017

  • This is a bug report
  • This is a feature request
  • I searched existing issues before opening this one

Expected behavior

i'm trying to deploy an elastic service across swarm cluster using compose file.
docker-compose starts elastic container normally, applying ulimits settings from .yml file.
but docker stack deploy ignores ulimits directive, so elasticsearch doesn't start at all.

.yml file is always the same, ether it docker-compose deploy or docker stack deploy

xx@xx:~/git/elk-compose-full$ docker-compose -f elastic-compose-single.yml up
WARNING: Some services (elasticsearch) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm.
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating network "elkcomposefull_esnet" with the default driver
Creating volume "elkcomposefull_data" with local driver
Creating elasticsearch ...
Creating elasticsearch ... done
Attaching to elasticsearch
elasticsearch    | [2017-08-29T09:15:52,331][INFO ][o.e.n.Node               ] [] initializing ...
elasticsearch    | [2017-08-29T09:15:52,397][INFO ][o.e.e.NodeEnvironment    ] [TibRwks] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/tmpl--vg-root)]], net usable_space [2.8gb], net total_space [7.2gb], spins? [possibly], types [ext4]
elasticsearch    | [2017-08-29T09:15:52,397][INFO ][o.e.e.NodeEnvironment    ] [TibRwks] heap size [494.9mb], compressed ordinary object pointers [true]
elasticsearch    | [2017-08-29T09:15:52,398][INFO ][o.e.n.Node               ] node name [TibRwks] derived from node ID [TibRwkssREK1G3d_32IsXg]; set [node.name] to override
elasticsearch    | [2017-08-29T09:15:52,399][INFO ][o.e.n.Node               ] version[5.5.2], pid[1], build[b2f0c09/2017-08-14T12:33:14.154Z], OS[Linux/4.4.0-92-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
elasticsearch    | [2017-08-29T09:15:52,399][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -Des.path.home=/usr/share/elasticsearch]
elasticsearch    | [2017-08-29T09:15:53,815][INFO ][o.e.p.PluginsService     ] [TibRwks] loaded module [aggs-matrix-stats]
elasticsearch    | [2017-08-29T09:15:53,815][INFO ][o.e.p.PluginsService     ] [TibRwks] loaded module [ingest-common]
elasticsearch    | [2017-08-29T09:15:53,815][INFO ][o.e.p.PluginsService     ] [TibRwks] loaded module [lang-expression]
elasticsearch    | [2017-08-29T09:15:53,815][INFO ][o.e.p.PluginsService     ] [TibRwks] loaded module [lang-groovy]
elasticsearch    | [2017-08-29T09:15:53,815][INFO ][o.e.p.PluginsService     ] [TibRwks] loaded module [lang-mustache]
elasticsearch    | [2017-08-29T09:15:53,815][INFO ][o.e.p.PluginsService     ] [TibRwks] loaded module [lang-painless]
elasticsearch    | [2017-08-29T09:15:53,815][INFO ][o.e.p.PluginsService     ] [TibRwks] loaded module [parent-join]
elasticsearch    | [2017-08-29T09:15:53,815][INFO ][o.e.p.PluginsService     ] [TibRwks] loaded module [percolator]
elasticsearch    | [2017-08-29T09:15:53,815][INFO ][o.e.p.PluginsService     ] [TibRwks] loaded module [reindex]
elasticsearch    | [2017-08-29T09:15:53,816][INFO ][o.e.p.PluginsService     ] [TibRwks] loaded module [transport-netty3]
elasticsearch    | [2017-08-29T09:15:53,816][INFO ][o.e.p.PluginsService     ] [TibRwks] loaded module [transport-netty4]
elasticsearch    | [2017-08-29T09:15:53,816][INFO ][o.e.p.PluginsService     ] [TibRwks] loaded plugin [ingest-geoip]
elasticsearch    | [2017-08-29T09:15:53,816][INFO ][o.e.p.PluginsService     ] [TibRwks] loaded plugin [ingest-user-agent]
elasticsearch    | [2017-08-29T09:15:53,816][INFO ][o.e.p.PluginsService     ] [TibRwks] loaded plugin [x-pack]
elasticsearch    | [2017-08-29T09:15:55,638][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/55] [Main.cc@128] controller (64 bit): Version 5.5.2 (Build 0d83940bb9d682) Copyright (c) 2017 Elasticsearch BV
elasticsearch    | [2017-08-29T09:15:55,666][INFO ][o.e.d.DiscoveryModule    ] [TibRwks] using discovery type [zen]
elasticsearch    | [2017-08-29T09:15:56,261][INFO ][o.e.n.Node               ] initialized
elasticsearch    | [2017-08-29T09:15:56,261][INFO ][o.e.n.Node               ] [TibRwks] starting ...
elasticsearch    | [2017-08-29T09:15:56,452][INFO ][o.e.t.TransportService   ] [TibRwks] publish_address {172.19.0.2:9300}, bound_addresses {0.0.0.0:9300}
elasticsearch    | [2017-08-29T09:15:56,466][INFO ][o.e.b.BootstrapChecks    ] [TibRwks] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
elasticsearch    | [2017-08-29T09:15:59,643][INFO ][o.e.c.s.ClusterService   ] [TibRwks] new_master {TibRwks}{TibRwkssREK1G3d_32IsXg}{ZAdw2rV1Q2-5ZuiP7PIhvg}{172.19.0.2}{172.19.0.2:9300}{ml.max_open_jobs=10, ml.enabled=true}, reason: zen-disco-elected-as-master ([0] nodes joined)
elasticsearch    | [2017-08-29T09:15:59,716][INFO ][o.e.h.n.Netty4HttpServerTransport] [TibRwks] publish_address {172.19.0.2:9200}, bound_addresses {0.0.0.0:9200}
elasticsearch    | [2017-08-29T09:15:59,716][INFO ][o.e.n.Node               ] [TibRwks] started
elasticsearch    | [2017-08-29T09:15:59,764][INFO ][o.e.g.GatewayService     ] [TibRwks] recovered [0] indices into cluster_state
elasticsearch    | [2017-08-29T09:15:59,978][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [TibRwks] successfully created .ml-state index template
elasticsearch    | [2017-08-29T09:15:59,990][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [TibRwks] successfully created .ml-meta index template
elasticsearch    | [2017-08-29T09:16:00,026][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [TibRwks] successfully created .ml-anomalies- index template
elasticsearch    | [2017-08-29T09:16:00,039][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [TibRwks] successfully created .ml-notifications index template
elasticsearch    | [2017-08-29T09:16:00,214][INFO ][o.e.l.LicenseService     ] [TibRwks] license [e1688e1c-cd69-49f4-9e39-fd2a621f2558] mode [trial] - valid

Actual behavior

docker stack deploy -c elastic-compose-single.yml elk

xx@xx:~/git/elk-compose-full$ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                                                 PORTS
q0u1pnz6zc2k        elk_elasticsearch   replicated          1/1                 docker.elastic.co/elasticsearch/elasticsearch:5.5.2   *:9200->9200/tcp

xx@xx:~/git/elk-compose-full$ docker service logs q0u1pnz6zc2k
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:52,929][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:52,931][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:52,931][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:52,931][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example:
elk_elasticsearch.1.a83hfcmmfma7@xx    |     # allow user 'elasticsearch' mlockall
elk_elasticsearch.1.a83hfcmmfma7@xx    |     elasticsearch soft memlock unlimited
elk_elasticsearch.1.a83hfcmmfma7@xx    |     elasticsearch hard memlock unlimited
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:52,931][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:53,042][INFO ][o.e.n.Node               ] [] initializing ...
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:53,153][INFO ][o.e.e.NodeEnvironment    ] [dT7I27E] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/tmpl--vg-root)]], net usable_space [2.8gb], net total_space [7.2gb], spins? [possibly], types [ext4]
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:53,153][INFO ][o.e.e.NodeEnvironment    ] [dT7I27E] heap size [494.9mb], compressed ordinary object pointers [true]
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:53,155][INFO ][o.e.n.Node               ] node name [dT7I27E] derived from node ID [dT7I27EwQOaQpekZF0mVsw]; set [node.name] to override
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:53,155][INFO ][o.e.n.Node               ] version[5.5.2], pid[1], build[b2f0c09/2017-08-14T12:33:14.154Z], OS[Linux/4.4.0-92-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:53,155][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -Des.path.home=/usr/share/elasticsearch]
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:54,608][INFO ][o.e.p.PluginsService     ] [dT7I27E] loaded module [aggs-matrix-stats]
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:54,608][INFO ][o.e.p.PluginsService     ] [dT7I27E] loaded module [ingest-common]
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:54,608][INFO ][o.e.p.PluginsService     ] [dT7I27E] loaded module [lang-expression]
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:54,608][INFO ][o.e.p.PluginsService     ] [dT7I27E] loaded module [lang-groovy]
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:54,608][INFO ][o.e.p.PluginsService     ] [dT7I27E] loaded module [lang-mustache]
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:54,609][INFO ][o.e.p.PluginsService     ] [dT7I27E] loaded module [lang-painless]
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:54,609][INFO ][o.e.p.PluginsService     ] [dT7I27E] loaded module [parent-join]
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:54,609][INFO ][o.e.p.PluginsService     ] [dT7I27E] loaded module [percolator]
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:54,610][INFO ][o.e.p.PluginsService     ] [dT7I27E] loaded module [reindex]
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:54,610][INFO ][o.e.p.PluginsService     ] [dT7I27E] loaded module [transport-netty3]
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:54,610][INFO ][o.e.p.PluginsService     ] [dT7I27E] loaded module [transport-netty4]
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:54,610][INFO ][o.e.p.PluginsService     ] [dT7I27E] loaded plugin [ingest-geoip]
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:54,610][INFO ][o.e.p.PluginsService     ] [dT7I27E] loaded plugin [ingest-user-agent]
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:54,611][INFO ][o.e.p.PluginsService     ] [dT7I27E] loaded plugin [x-pack]
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:56,448][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/55] [Main.cc@128] controller (64 bit): Version 5.5.2 (Build 0d83940bb9d682) Copyright (c) 2017 Elasticsearch BV
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:56,479][INFO ][o.e.d.DiscoveryModule    ] [dT7I27E] using discovery type [zen]
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:57,116][INFO ][o.e.n.Node               ] initialized
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:57,116][INFO ][o.e.n.Node               ] [dT7I27E] starting ...
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:57,314][INFO ][o.e.t.TransportService   ] [dT7I27E] publish_address {10.0.0.2:9300}, bound_addresses {0.0.0.0:9300}
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:57,325][INFO ][o.e.b.BootstrapChecks    ] [dT7I27E] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
elk_elasticsearch.1.a83hfcmmfma7@xx    | ERROR: [1] bootstrap checks failed
elk_elasticsearch.1.a83hfcmmfma7@xx    | [1]: memory locking requested for elasticsearch process but memory is not locked
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:57,333][INFO ][o.e.n.Node               ] [dT7I27E] stopping ...
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:57,384][INFO ][o.e.n.Node               ] [dT7I27E] stopped
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:57,384][INFO ][o.e.n.Node               ] [dT7I27E] closing ...
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:57,408][INFO ][o.e.n.Node               ] [dT7I27E] closed
elk_elasticsearch.1.a83hfcmmfma7@xx    | [2017-08-29T09:08:57,412][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started

Steps to reproduce the behavior

elastic-compose-single.yml:

version: '3'

services:

    elasticsearch:
        container_name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:5.5.2
        volumes:
            - /etc/localtime:/etc/localtime:ro
            - data:/usr/share/elasticsearch/data
        environment:
            - xpack.security.enabled=false
            - xpack.monitoring.enabled=false
            - cluster.name=elastic-cluster
            - bootstrap.memory_lock=true
            - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
        ports:
            - 9200:9200
        ulimits:
            memlock:
              soft: -1
              hard: -1
        deploy:
            resources:
              limits:
                memory: 1000M
        networks:
            - esnet

volumes:
    data:
        driver: local

networks:
    esnet:
xx@xx:~/git/elk-compose-full$ docker stack deploy -c elastic-compose-single.yml elk
Ignoring deprecated options:

container_name: Setting the container name is not supported.

Creating network elk_esnet
Creating service elk_elasticsearch
xx@xx:~/git/elk-compose-full$ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                                                 PORTS
q0u1pnz6zc2k        elk_elasticsearch   replicated          1/1                 docker.elastic.co/elasticsearch/elasticsearch:5.5.2   *:9200->9200/tcp
xx@xx:~/git/elk-compose-full$ docker service logs q0u1pnz6zc2k
elk_elasticsearch.1.a83hfcmmfma7@xxx    | [2017-08-29T09:08:52,929][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
elk_elasticsearch.1.a83hfcmmfma7@xxx    | [2017-08-29T09:08:52,931][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
elk_elasticsearch.1.a83hfcmmfma7@xxx    | [2017-08-29T09:08:52,931][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
elk_elasticsearch.1.a83hfcmmfma7@xxx    | [2017-08-29T09:08:52,931][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example:
elk_elasticsearch.1.a83hfcmmfma7@xxx    |     # allow user 'elasticsearch' mlockall
elk_elasticsearch.1.a83hfcmmfma7@xxx    |     elasticsearch soft memlock unlimited
elk_elasticsearch.1.a83hfcmmfma7@xxx    |     elasticsearch hard memlock unlimited
elk_elasticsearch.1.a83hfcmmfma7@xxx    | [2017-08-29T09:08:52,931][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.

Output of docker version:

Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:10:54 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:10:54 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 17.05.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 11
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: lrsfbttb3qsasbb04fhhawru3
 Is Manager: true
 ClusterID: l14zv2dzph783ccusuwg622sp
 Managers: 1
 Nodes: 4
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 192.168.248.40
 Manager Addresses:
  192.168.248.40:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-92-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.953GiB
Name: xxx
ID: FZCA:5HIY:FAYG:JV3Y:Q3NG:M327:GKCD:TZX4:WLX3:MDKI:HX5Z:CRHV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.)

docker host on vmware guest esxi 6

@dnephin
Copy link

dnephin commented Aug 29, 2017

Thanks for the report. It looks like we forgot to add ulimits to the list of unsupported fields. I've opened a PR to add it (docker/cli#482).

Unfortunately it can't be supported until it's supported by swarmkit. The relevant issue is here: moby/moby#25209

Going to close this issue since we already have a tracking issue.

@00willo
Copy link

00willo commented Dec 3, 2022

Looks like this is now resolved in 20.10.0. Haven't tested it yet.

docker/cli#2712

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants