1<!doctype html public "-//W3C//DTD HTML 4.01 Transitional//EN" 2 "http://www.w3.org/TR/html4/loose.dtd"> 3 4<html> 5 6<head> 7 8<title>Postfix OpenLDAP LMDB Howto</title> 9 10<meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> 11 12</head> 13 14<body> 15 16<h1><img src="postfix-logo.jpg" width="203" height="98" ALT="">Postfix OpenLDAP LMDB Howto</h1> 17 18<hr> 19 20<h2>Introduction</h2> 21 22<p> Postfix uses databases of various kinds to store and look up 23information. Postfix databases are specified as "type:name". OpenLDAP 24LMDB (called "LMDB" from here on) implements the Postfix database 25type "lmdb". The name of a Postfix LMDB database is the name of 26the database file without the ".lmdb" suffix. </p> 27 28<p> This document describes: </p> 29 30<ul> 31 32<li> <p> <a href="#with_lmdb">Building Postfix with LMDB support</a>. 33</p> 34 35<li> <p> <a href="#configure">Configuring LMDB settings</a>. </p> 36 37<li> <p> <a href="#locking">Using LMDB maps with non-Postfix programs</a>. </p> 38 39<li> <p> <a href="#supported"> Required minimum LMDB patchlevel</a>. </p> 40 41<li> <p> <a href="#credits"> Credits</a>. </p> 42 43</ul> 44 45<h2><a name="with_lmdb">Building Postfix with LMDB support</a></h2> 46 47<p> Postfix normally does not enable LMDB support. To 48build Postfix with LMDB support, use something like: </p> 49 50<blockquote> 51<pre> 52% make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \ 53 AUXLIBS="-L/usr/local/lib -llmdb" 54% make 55</pre> 56</blockquote> 57 58<p> Solaris may need this: </p> 59 60<blockquote> 61<pre> 62% make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \ 63 AUXLIBS="-R/usr/local/lib -L/usr/local/lib -llmdb" 64% make 65</pre> 66</blockquote> 67 68<p> The exact pathnames depend on how LMDB was installed. </p> 69 70<p> When building Postfix fails with: </p> 71 72<blockquote> 73<pre> 74undefined reference to `pthread_mutexattr_destroy' 75undefined reference to `pthread_mutexattr_init' 76undefined reference to `pthread_mutex_lock' 77</pre> 78</blockquote> 79 80<p> Add the "-lpthread" library to the "make makefiles" command. </p> 81 82<blockquote> 83<pre> 84% make makefiles .... AUXLIBS="... -lpthread" 85</pre> 86</blockquote> 87 88<h2><a name="configure">Configuring LMDB settings</a></h2> 89 90<p> Postfix provides one configuration parameter that controls 91LMDB database behavior. </p> 92 93<ul> 94 95<li> <p> <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> (default: 16777216). This setting specifies 96the initial LMDB database size limit in bytes. Each time a database 97becomes "full", its size limit is doubled. The maximum size is the 98largest signed integer value of "long". </p> 99 100</ul> 101 102<h2> <a name="locking">Using LMDB maps with non-Postfix programs</a> </h2> 103 104<p> Programs that use LMDB's built-in locking protocol will corrupt 105a Postfix LMDB database or will read garbage. </p> 106 107<p> Postfix does not use LMDB's built-in locking protocol, because 108that would require world-writable lockfiles, and would violate 109Postfix security policy. Instead, Postfix uses external locks based 110on fcntl(2) to prevent writers from corrupting the database, and 111to prevent readers from receiving garbage. </p> 112 113<p> See <a href="lmdb_table.5.html">lmdb_table(5)</a> for a detailed description of the locking 114protocol that all programs must use when they access a Postfix LMDB 115database. </p> 116 117<h2> <a name="supported"> Required minimum LMDB patchlevel </a> </h2> 118 119<p> Currently, Postfix requires LMDB 0.9.11 or later. The required 120minimum LMDB patchlevel has evolved over time, as the result of 121Postfix deployment experience: </p> 122 123<ul> 124 125<li> <p> LMDB 0.9.11 allows Postfix daemons to log an LMDB error 126message, instead of falling out of the sky without any notification. 127</p> 128 129<li> <p> LMDB 0.9.10 closes an information leak where LMDB was 130writing up to 4-kbyte chunks of uninitialized heap memory to the 131database. This would persist information that was not meant to be 132persisted, or share information that was not meant to be shared. 133</p> 134 135<li> <p> LMDB 0.9.9 allows Postfix to use external (fcntl()-based) 136locks, instead of having to use world-writable LMDB lock files, 137violating the Postfix security model in multiple ways. </p> 138 139<li> <p> LMDB 0.9.8 allows Postfix to recover from a "database full" 140error without having to close the database. This version adds support 141to update the database size limit on-the-fly. This is necessary 142because Postfix database sizes vary with mail server load. </p> 143 144<li> <p> LMDB 0.9.7 allows the <a href="postmap.1.html">postmap(1)</a> and <a href="postalias.1.html">postalias(1)</a> commands 145to use a bulk-mode transaction larger than the amount of physical 146memory. This is necessary because LMDB supports databases larger 147than physical memory. </p> 148 149</ul> 150 151<h2> <a name="credits"> Credits</a> </h2> 152 153<ul> 154 155<li> <p> Howard Chu contributed the initial Postfix dict_lmdb driver. 156</p> 157 158<li> <p> Wietse Venema wrote an abstraction layer (slmdb) that 159behaves more like Berkeley DB, NDBM, etc. This layer automatically 160retries an LMDB request when a database needs to be resized, or 161after a database was resized by a different process. </p> 162 163<li> <p> Howard and Wietse went through many iterations with changes 164to both LMDB and Postfix, with input from Viktor Dukhovni. </p> 165 166</ul> 167 168<!-- 169 170<h2><a name="limitations">Unexpected failure modes of Postfix LMDB 171databases. </a> </h2> 172 173<p> As documented below, conversion to LMDB introduces a number of 174failure modes that don't exist with other Postfix databases. Some 175failure modes have been eliminated in the course of time. 176The writeup below reflects the status as of LMDB 0.9.9. </p> 177 178--> 179 180<!-- 181 182<p> <strong>Unexpected "Permission denied" errors. </strong></p> 183 184<dl> 185 186<dt> Problem: </dt> <dd> <p> A world-readable LMDB database cannot 187be opened by a process with a UID that differs from the database 188file owner, even when an attempt is made to open the database 189read-only. This problem does not exist with other Postfix databases. 190</p> </dd> 191 192<dt> Background: </dt> <dd> <p> The LMDB implementation requires 193write access to maintain read locks, and perhaps for other purposes. 194</p> </dd> 195 196<dt> Solution: </dt> <dd> <p> Consider using <a href="CDB_README.html">cdb</a>: to manage root-owned 197databases under the root-owned <tt>/etc</tt> or <a href="postconf.5.html#config_directory">config_directory</a> 198(default: <tt>/etc/postfix</tt>) such as <a href="access.5.html">access(5)</a>, <a href="virtual.5.html">virtual(5)</a>, 199<a href="transport.5.html">transport(5)</a>. Support to create LMDB databases is available only 200for unprivileged Postfix daemon processes such as <a href="postscreen.8.html">postscreen(8)</a>, 201<a href="tlsmgr.8.html">tlsmgr(8)</a> and <a href="verify.8.html">verify(8)</a> that manage postfix-owned databases under 202the postfix-owned <a href="postconf.5.html#data_directory">data_directory</a> (default: <tt>/var/lib/postfix</tt>). 203</p> </dd> 204 205</dl> 206 207--> 208 209<!-- 210 211<p> <strong>Unexpected "readers full" errors. </strong></p> 212 213<dl> 214 215<dt> Problem: </dt> <dd> <p> Under heavy load, database read 216operations fail with MDB_READERS_FULL errors. This problem does not 217exist with other Postfix databases. </p> </dd> 218 219<dt> Background: </dt> <dd> <p> The LMDB implementation enforces a 220hard limit on the number of simultaneous read requests for the same 221database environment. This limit must be specified in advance with 222the lmdb_max_readers configuration parameter. </p> </dd> 223 224<dt> Mitigation: </dt> <dd> <p> Postfix logs a warning suggesting 225that the lmdb_max_readers parameter value be increased, and retries 226the failed operation for a limited number of times while running 227with reduced performance. </p> </dd> 228 229<dt> Prevention: </dt> <dd> <p> Monitor your LMDB files for 230MDB_READERS_FULL errors. After making the necessary adjustments, 231restart Postfix. </p> </dd> 232 233</dl> 234 235--> 236 237<!-- 238 239<p> <strong>Unexpected <a href="postmap.1.html">postmap(1)</a>/<a href="postalias.1.html">postalias(1)</a> "database full" 240errors. </strong></p> 241 242<dl> 243 244<dt> Problem: </dt> <dd> <p> The "postmap <a href="lmdb_table.5.html">lmdb</a>:filename" command 245fails with an MDB_TXN_FULL error. This problem does not exist with 246other Postfix databases. </p> </dd> 247 248<dt> Background: </dt> 249 250<dd> 251 252<p> The LMDB implementation has a hard limit on the total transaction 253size. This limit is independent of the LMDB database size. Therefore, 254the problem cannot be resolved by increasing the <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> 255value. </p> 256 257<p> This symptom is indicative of a flawed design. All LMDB data 258structures should share the same storage pool so that they can scale 259with the database size, and so that all "out of storage" errors are 260resolved by increasing the database size. </p> </dd> 261 262--> 263 264<!-- 265 266<p> Problem: </dt> <dd> <p> The "postmap <a href="lmdb_table.5.html">lmdb</a>:filename" command 267fails with an MDB_MAP_FULL error. This problem does not exist with 268other Postfix databases. </p> </dd> 269 270<dl> 271 272<dt> Background: </dt> 273 274<dd> 275 276<p> LMDB databases have a hard size limit (configured with the 277<a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> configuration parameter). </p> 278 279<p> When executing "postmap <a href="lmdb_table.5.html">lmdb</a>:filename", the Postfix LMDB database 280client stores the new data in a transaction which takes up space 281in addition to the existing data, and commits the transaction when 282it closes the database. Only then can the space for old data be 283reused. </p> 284 285</dd> 286 287<dt> Impact: </dt> <dd> <p> This failure does not affect Postfix 288availability, because the old data still exists in the database. 289</p> </dd> 290 291<dt> Mitigation: </dt> <dd> 292 293<p> When the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a> command fails with an 294MDB_MAP_FULL error, it expands the database file size to the current 295LMDB map size limit before terminating. </p> 296 297<p> Next, when you re-run the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a> command, 298it discovers that the LMDB file is larger than <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a>/3, 299logs a warning, and uses a larger LMDB map size limit instead: </p> 300 301<p> <tt> warning: <i>filename</i>.<a href="lmdb_table.5.html">lmdb</a>: file size 15024128 ≥ 302(lmdb map size limit 16777216)/3<br> warning: <i>filename</i>.<a href="lmdb_table.5.html">lmdb</a>: 303using map size limit 45072384</tt> </p> 304 305<p> By repeating the two steps above you can automate recovery and 306avoid the need for human intervention. Just repeat "postmap 307<a href="lmdb_table.5.html">lmdb</a>:filename" (up to some limit). After each failure it will use 308a 3x larger size limit, and eventually the "database full" error 309should disappear. This fails only when the disk is full or when 310the LMDB map size limit would exceed the memory address space size 311limit. </p> 312 313<dt> Prevention: </dt> <dd> <p> Monitor your LMDB files and make 314sure that in <a href="postconf.5.html">main.cf</a>, <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> > 3x the largest LMDB file 315size. </p> </dd> </dl> 316 317</dl> 318 319--> 320 321<!-- 322 323<p> <strong>Unexpected Postfix daemon "database full" errors. 324</strong></p> 325 326<dl> 327 328<dt> Problem: </dt> <dd> <p> Postfix daemon programs fail with 329"database full" errors, such as <a href="postscreen.8.html">postscreen(8)</a>, <a href="tlsmgr.8.html">tlsmgr(8)</a> or <a href="verify.8.html">verify(8)</a>. 330This problem does not exist with other Postfix databases. </p> 331</dd> 332 333<dt> Impact: </dt> <dd> <p> This failure temporarily affects Postfix 334availability. The daemon restarts automatically and tries to open 335the database again as described next. </p> </dd> 336 337<dt> Mitigation: </dt> <dd> <p> When a Postfix daemon opens an LMDB 338file larger than <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a>/3, it logs a warning and uses a 339larger size limit instead: </p> 340 341<p> <tt> warning: <i>filename</i>.<a href="lmdb_table.5.html">lmdb</a>: file size 15024128 ≥ 342(lmdb map size limit 16777216)/3 <br>warning: <i>filename</i>.<a href="lmdb_table.5.html">lmdb</a>: 343using map size limit 45072384</tt> </p> 344 345<p> This can be used to automate recovery and avoid the need for 346human intervention. Each time the daemon runs into a "database full" 347error, it restarts and uses a 3x larger size limit. The "database 348full" error will disappear, at least for a while. </p> 349 350<dt> Prevention: </dt> <dd> <p> Monitor your LMDB files and make 351sure that <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> > 3x the largest LMDB file size. </p> 352</dd> </dl> 353 354--> 355 356<!-- 357 358<p> <strong>Non-obvious recovery with <a href="postmap.1.html">postmap(1)</a>, <a href="postalias.1.html">postalias(1)</a>, or 359<a href="tlsmgr.8.html">tlsmgr(8)</a> from a corrupted database. </strong></p> 360 361<dl> 362 363<dt> Problem: </dt> <dd> <p> A corrupted LMDB database can't be 364rebuilt simply by re-running <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a>, or by 365waiting until a <a href="tlsmgr.8.html">tlsmgr(8)</a> daemon restarts. This problem does not 366exist with other Postfix databases. </p> </dd> 367 368<dt> Background: </dt> <dd> <p> The Postfix LMDB database client 369does not truncate the database file. Instead it attempts to create 370a transaction for a "drop" request plus subsequent "store" requests. 371That is obviously not possible with a corrupted database file. </p> 372</dd> 373 374<dt> Impact: </dt> <dd> <p> Postfix does not process mail until 375someone fixes the problem. </p> </dd> 376 377<dt> Recovery: </dt> <dd> <p> First delete the ".lmdb" file by hand. 378Then rebuild the file with the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a> 379command if the file was created with those commands, or restart 380postfix daemons if the file is maintained by <a href="tlsmgr.8.html">tlsmgr(8)</a>. 381</p> </dd> 382 383<dt> Prevention: </dt> <dd> 384 385<p> Arrange your file systems such that they never run out of free 386space. </p> 387 388<p> Use ECC memory to detect and correct silent corruption of 389in-memory file system data and metadata. </p> 390 391<p> Use a file system such as ZFS to detect and correct silent 392corruption of on-disk file system data and metadata. DO NOT 393use ZFS on systems without ECC memory error correction. </p> 394 395</dd> </dl> 396 397--> 398 399</body> 400 401</html> 402
