-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathsharedmap.html
210 lines (197 loc) · 14.4 KB
/
sharedmap.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
<!DOCTYPE html>
<html>
<head>
<meta charset='utf-8'>
<meta http-equiv="X-UA-Compatible" content="chrome=1">
<meta content="True" name="HandheldFriendly">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0">
<meta name="viewport" content="width=device-width">
<meta name="description" content="nginx-clojure.github.io : Nginx module for embedding Clojure / Java / Groovy programs, typically those Ring based handlers">
<link rel="stylesheet" type="text/css" media="screen" href="stylesheets/stylesheet.css">
<!-- bootstrap cdn -->
<!-- Latest compiled and minified CSS -->
<link rel="stylesheet" href="//netdna.bootstrapcdn.com/bootstrap/3.0.0/css/bootstrap.min.css">
<!-- Optional theme -->
<link rel="stylesheet" href="//netdna.bootstrapcdn.com/bootstrap/3.0.0/css/bootstrap-theme.min.css">
<script src="//ajax.aspnetcdn.com/ajax/jQuery/jquery-1.11.0.min.js"></script>
<!-- Latest compiled and minified JavaScript -->
<script src="//netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<script src="javascripts/fix.js"></script>
<link rel="stylesheet" type="text/css" media="screen" href="stylesheets/my.css">
<title>Shared Map & Session Store@Nginx-Clojure</title>
</head>
<body>
<div id="header">
<!-- HEADER -->
<nav id="header_wrap" class="" role="navigation">
<header id="header_logo_title" class="inner container">
<ul class="external-link-buttons">
<li>
<a id="forkme_banner" href="https://github.com/nginx-clojure/nginx-clojure" target="_blank">View on GitHub</a>
</li>
<!-- <li>
<a id="google_group_banner" href="https://groups.google.com/forum/#!forum/nginx-clojure" target="_blank">Google Group</a>
</li> -->
</ul>
<div id="project_logo">
<h1 id="project_title">nginx-clojure</h1>
<ul class="social-buttons">
<li>
<iframe class="github-btn" src="github-btn.html?user=nginx-clojure&repo=nginx-clojure&type=watch&count=true" allowtransparency="true" frameborder="0" scrolling="0" width="102px" height="20px"></iframe>
</li>
<li>
<iframe class="github-btn" src="github-btn.html?user=nginx-clojure&repo=nginx-clojure&type=fork&count=true" allowtransparency="true" frameborder="0" scrolling="0" width="100px" height="20px"></iframe>
</li>
</ul></div>
<h6 id="project_tagline">Nginx module for embedding Clojure / Java / Groovy programs, typically those Ring based handlers</h6>
</header>
<div class="btn-group" style="border-top-width: 1px;border-top-color: #fff;">
<a href="index.html" class="btn btn-sample btn-md"><span class="glyphicon glyphicon-home"></span> Home</a>
<a href="quickstart.html" class="btn btn-sample btn-md"><span class="glyphicon glyphicon-flash"></span> Quick Start</a>
<a href="downloads.html" class="btn btn-sample btn-md"><span class="glyphicon glyphicon-download-alt"></span> Downloads</a>
<a href="installation.html" class="btn btn-sample btn-md"><span class="glyphicon glyphicon-hdd"></span> Installation</a>
<div class="btn-group">
<button type="button" class="btn btn-sample btn-md dropdown-toggle" data-toggle="dropdown" id="docDropDownBtn"><span class="glyphicon glyphicon-cog"></span> Configuration <span class="caret"></span></button>
<ul class="dropdown-menu" role="menu">
<li><a href="configuration.html#user-content-21-jvm-path--class-path--other-jvm-options"><span class="glyphicon glyphicon-cog"></span> JVM Path,Class Path & Other JVM Options</a></li>
<li><a href="configuration.html#user-content-22-initialization-handler-for-nginx-worker"><span class="glyphicon glyphicon-cog"></span> Initialization Handler for nginx worker</a></li>
<li><a href="configuration.html#user-content-23-content-ring-handler-for-location"><span class="glyphicon glyphicon-cog"></span> Content Ring Handler for Location</a></li>
<li><a href="configuration.html#user-content-24-chose--coroutine-based-socket-or-asynchronous-socketchannel-or-thread-pool-for-slow-io-operations"><span class="glyphicon glyphicon-cog"></span> Coroutine/Asynchronous Client Channel/Thread Pool </a></li>
<li><a href="configuration.html#user-content-25-nginx-rewrite-handler"><span class="glyphicon glyphicon-cog"></span> Nginx Rewrite Handler</a></li>
<li><a href="configuration.html#user-content-26-nginx-access-handler"><span class="glyphicon glyphicon-cog"></span> Nginx Access Handler</a></li>
<li><a href="configuration.html#user-content-27-nginx-header-filter"><span class="glyphicon glyphicon-cog"></span> Nginx Header Filter</a></li>
<li><a href="configuration.html#user-content-28-nginx-body-filter"><span class="glyphicon glyphicon-cog"></span> Nginx Body Filter</a></li>
<li><a href="configuration.html#user-content-29-nginx-log-handler"><span class="glyphicon glyphicon-cog"></span> Nginx Log Handler</a></li>
</ul>
</div>
<div class="btn-group">
<button type="button" class="btn btn-sample btn-md dropdown-toggle" data-toggle="dropdown" id="docDropDownBtn"><span class="glyphicon glyphicon-book"></span> Documents <span class="caret"></span></button>
<ul class="dropdown-menu" role="menu">
<!-- <li><a href="quickstart.html"><span class="glyphicon glyphicon-flash"></span> Quick Start</a></li> -->
<li><a href="directives.html"><span class="glyphicon glyphicon-th-list text-warning"></span> Directives Reference</a></li>
<li><a href="embed.html"><span class="glyphicon glyphicon-gift"></span> Embedding Nginx-Clojure into A standard App</a></li>
<li><a href="more.html#user-content-34-server-channel-for-long-polling--server-sent-events-sse"><span class="glyphicon glyphicon-envelope"></span> Server Channel for Long Polling & Server Sent Events</a></li>
<li><a href="subpub.html"><span class="glyphicon glyphicon-envelope text-warning"></span> Pub/Sub Among Nginx Worker Processes</a></li>
<li><a href="sharedmap.html"><span class="glyphicon glyphicon-envelope text-warning"></span> Shared Map & Session Store</a></li>
<li><a href="more.html#user-content-36-asynchronous-client-channel"><span class="glyphicon glyphicon-road"></span> Asynchronous Client Channel</a></li>
<li><a href="more.html#user-content-37--about-logging"><span class="glyphicon glyphicon-th-list"></span> About Logging</a></li>
<li><a href="more.html#user-content-38--sever-side-websocket"><span class="glyphicon glyphicon-th-list"></span> Sever Side WebSocket</a></li>
<li><a href="more.html#user-content-39--java-standard-restful-web-services-with-jersey"><span class="glyphicon glyphicon-th-list"></span> Java standard RESTful web services with Jersey</a></li>
<li><a href="more.html#user-content-310-embeding-tomcat"><span class="glyphicon glyphicon-th-list"></span> Embeding Tomcat</a></li>
<li><a href="more.html#user-content-311-more-about-nginx-worker-process"><span class="glyphicon glyphicon-th-list"></span> More about Nginx Worker Process</a></li>
<li><a href="more.html"><span class="glyphicon glyphicon-th-list"></span> More about Nginx-Clojure</a></li>
<li><a href="api/"><span class="glyphicon glyphicon-book text-warning"></span> API Reference (Clojure)</a></li>
</ul>
</div>
<a href="userfullLinks.html" class="btn btn-sample btn-md"><span class="glyphicon glyphicon-share-alt"></span> Useful Links</a>
</div>
</nav>
</div>
<!-- MAIN CONTENT -->
<div id="main_content_wrap" class="outer">
<section id="main_content" class="inner">
<h1>
<a id="user-content-shared-map--session-store" class="anchor" href="#shared-map--session-store" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Shared Map & Session Store</h1>
<h2>
<a id="user-content-shared-map" class="anchor" href="#shared-map" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Shared Map</h2>
<p>Shared map is used to share data among nginx worker processes without any other external services
(e.g. redis, memorycached ) or libraries (e.g. SharedHashMap/Chronicle-Map, nginx lua shared dic).
So far it has two implementations: tiny map & hash map both of which use MurmurHash3 32-bit to
generate hash code. The key/value of shared hash map can be <code>int</code>,<code>long</code>,<code>String</code>, <code>byte array</code>.</p>
<p><strong>limitation</strong></p>
<ol>
<li>Tiny Map
<ul>
<li>entries number limit: 2^31=2.14Billions</li>
<li>total space limit : 4G (64-bit) / 2G (32-bit)</li>
<li>key size limit: 16M</li>
<li>value size limit: 4G (64-bit) / 2G (32-bit)</li>
<li>entry structure cost: 24 Bytes</li>
<li>table structure cost: entries x 4 Bytes</li>
</ul>
</li>
<li>Hash Map
<ul>
<li>entries number limit: 2^63 (64-bit) / 2^31 (32-bit)</li>
<li>total space limit : OS limit</li>
<li>key size limit: OS limit</li>
<li>value size limit: OS limit</li>
<li>entry structure cost: 40 Bytes (64-bit) / 28 Bytes (32-bit)</li>
<li>table structure cost: entries x 8 Bytes (64-bit) / entries x 4 (32-bit)</li>
</ul>
</li>
</ol>
<p>But note that if needed memory size is less than half of OS page size the real allocated size of nginx slab only can be
2^3 = 8, 2^4 = 16, 2^5 = 32,..., 2^(ngx_pagesize_shift - 1).So on 64-bit OS entry structure size of tiny map really
uses 32Bytes hash map uses 64Bytes. We'll do some optimize work in the future versions.</p>
<p>Neither tiny map nor hash map will rehash entries because both of them will have a constant number of buckets once they are initialized.</p>
<pre><code>${number of buckets} = round_up_to_power_of_2( ${entries} * 0.75 )
</code></pre>
<p><strong>Optimization for int/long</strong></p>
<p>Some optimization have been done about java int/long key/value, int/long key/value and they won't allocate additional memory
because them are stored in the entry structure itself.
e.g. in a hash map for a java long key (64bits) we will store it into</p>
<ol>
<li>entry.key (64-bit OS) or</li>
<li>entry.key & entry.ksize (32-bit OS) because size of java long is constant we can reuse entry.ksize.</li>
</ol>
<p>viz. use c code</p>
<div class="highlight highlight-source-c"><pre>
*((<span class="pl-c1">uint64_t</span> *)(<span class="pl-k">void</span>*)&entry->key) = ${64bits java <span class="pl-k">long</span> value};
</pre></div>
<p><strong>Example</strong></p>
<p>Here's an example to use shared map to count uri access times.</p>
<p>In nginx.conf</p>
<div class="highlight highlight-source-nginx"><pre> <span class="pl-k">shared_map</span> uri_access_counters tinymap?space=1m&entries=8096;</pre></div>
<ul>
<li><strong>clojure</strong></li>
</ul>
<div class="highlight highlight-source-clojure"><pre>
<span class="pl-c"><span class="pl-c">;</span>; be friendly to embeded nginx-clojure where we maybe def shared map before server starts</span>
(<span class="pl-k">def</span> <span class="pl-e">dsmap</span> (<span class="pl-en">delay</span> (<span class="pl-en">nginx.clojure.clj.ClojureSharedHashMap.</span> <span class="pl-s"><span class="pl-pds">"</span>uri_access_counters<span class="pl-pds">"</span></span>)))
<span class="pl-c"><span class="pl-c">;</span>; if uri not exists set 1 otherwise increase its count.</span>
(<span class="pl-k">when</span> (<span class="pl-en">zero?</span> (<span class="pl-en">.putIntIfAbsent</span> @dsmap uri (<span class="pl-en">int</span> <span class="pl-c1">1</span>)))
(<span class="pl-en">.atomicAddInt</span> @dsmap uri (<span class="pl-en">int</span> <span class="pl-c1">1</span>)))</pre></div>
<ul>
<li><strong>Java</strong></li>
</ul>
<div class="highlight highlight-source-java"><pre><span class="pl-c"><span class="pl-c">//</span> get the shared map. Mostly we do this in a handler</span>
<span class="pl-c"><span class="pl-c">//</span> (e.g. jvm init handler, content handler) 's constructor.</span>
<span class="pl-smi">NginxSharedHashMap</span> smap <span class="pl-k">=</span> <span class="pl-smi">NginxSharedHashMap</span><span class="pl-k">.</span>build(<span class="pl-s"><span class="pl-pds">"</span>uri_access_counters<span class="pl-pds">"</span></span>);
<span class="pl-c"><span class="pl-c">//</span>if uri not exists set 1 otherwise increase its count.</span>
<span class="pl-c"><span class="pl-c">//</span>putIntIfAbsent is a faster version of putIfAbsent for int value</span>
<span class="pl-k">if</span> (smap<span class="pl-k">.</span>putIntIfAbsent(uri, <span class="pl-c1">1</span>) <span class="pl-k">!=</span> <span class="pl-c1">0</span>) {
smap<span class="pl-k">.</span>atomicAddInt(uri, <span class="pl-c1">1</span>);
}</pre></div>
<h2>
<a id="user-content-shared-map-based-ring-session-store" class="anchor" href="#shared-map-based-ring-session-store" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Shared Map Based Ring Session Store</h2>
<p>When worker_processes > 1 in nginx.conf, we can not use the default in-memory session store
because there're more than one JVM instances and requests from the same session perhaps
will be handled by different JVM instances. We can try cookie store, or shared map based ring session store
and if we use redis to shared sessions we can try <a href="https://github.com/ptaoussanis/carmine">carmine-store</a> or
<a href="https://github.com/wuzhe/clj-redis-session">redis session store</a>.
Create a shared map based ring session store is very simple. e.g.</p>
<p>In nginx.conf</p>
<div class="highlight highlight-source-nginx"><pre><span class="pl-k">shared_map</span> my-session-store tinymap?space=10m&entries=1024;</pre></div>
<ul>
<li><strong>Clojure</strong></li>
</ul>
<div class="highlight highlight-source-clojure"><pre>(<span class="pl-k">def</span> <span class="pl-e">nginx.clojure.session</span>/<span class="pl-e">my-session-store</span> (<span class="pl-en">shared-map-store</span> <span class="pl-s"><span class="pl-pds">"</span>my-session-store<span class="pl-pds">"</span></span>))</pre></div>
</section>
</div>
<div id="footer">
<!-- FOOTER -->
<div id="footer_wrap" class="outer">
<footer class="inner">
<p>
Powered by <a href="http://getbootstrap.com/">Twitter Bootstrap</a>
</p>
</footer>
</div>
</div>
<script type="text/javascript">
//$("a[href^='http']").attr("target","_blank");
//setTimeout("fixNavbarIssue();", 1000);
</script>
</body>
</html>