1<!doctype html public "-//W3C//DTD HTML 4.01 Transitional//EN" 2 "http://www.w3.org/TR/html4/loose.dtd"> 3 4<html> 5 6<head> 7 8<title>Postfix OpenLDAP LMDB Howto</title> 9 10<meta http-equiv="Content-Type" content="text/html; charset=utf-8"> 11 12</head> 13 14<body> 15 16<h1><img src="postfix-logo.jpg" width="203" height="98" ALT="">Postfix OpenLDAP LMDB Howto</h1> 17 18<hr> 19 20<h2>Introduction</h2> 21 22<p> Postfix uses databases of various kinds to store and look up 23information. Postfix databases are specified as "type:name". OpenLDAP 24LMDB (called "LMDB" from here on) implements the Postfix database 25type "lmdb". The name of a Postfix LMDB database is the name of 26the database file without the ".lmdb" suffix. </p> 27 28<p> This document describes: </p> 29 30<ul> 31 32<li> <p> <a href="#with_lmdb">Building Postfix with LMDB support</a>. 33</p> 34 35<li> <p> <a href="#configure">Configuring LMDB settings</a>. </p> 36 37<li> <p> <a href="#locking">Using LMDB maps with non-Postfix programs</a>. </p> 38 39<li> <p> <a href="#supported"> Required minimum LMDB patchlevel</a>. </p> 40 41<li> <p> <a href="#credits"> Credits</a>. </p> 42 43</ul> 44 45<h2><a name="with_lmdb">Building Postfix with LMDB support</a></h2> 46 47<p> Postfix normally does not enable LMDB support. To 48build Postfix with LMDB support, use something like: </p> 49 50<blockquote> 51<pre> 52% make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \ 53 AUXLIBS_LMDB="-L/usr/local/lib -llmdb" 54% make 55</pre> 56</blockquote> 57 58<p> If your LMDB shared library is in a directory that the RUN-TIME 59linker does not know about, add a "-Wl,-R,/path/to/directory" option after 60"-llmdb". </p> 61 62<p> Postfix versions before 3.0 use AUXLIBS instead of AUXLIBS_LMDB. 63With Postfix 3.0 and later, the old AUXLIBS variable still supports 64building a statically-loaded LMDB database client, but only the new 65AUXLIBS_LMDB variable supports building a dynamically-loaded or 66statically-loaded LMDB database client. </p> 67 68<blockquote> 69 70<p> Failure to use the AUXLIBS_LMDB variable will defeat the purpose 71of dynamic database client loading. Every Postfix executable file 72will have LMDB database library dependencies. And that was exactly 73what dynamic database client loading was meant to avoid. </p> 74 75</blockquote> 76 77 78<p> Solaris may need this: </p> 79 80<blockquote> 81<pre> 82% make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \ 83 AUXLIBS_LMDB="-R/usr/local/lib -L/usr/local/lib -llmdb" 84% make 85</pre> 86</blockquote> 87 88<p> The exact pathnames depend on how LMDB was installed. </p> 89 90<p> When building Postfix fails with: </p> 91 92<blockquote> 93<pre> 94undefined reference to `pthread_mutexattr_destroy' 95undefined reference to `pthread_mutexattr_init' 96undefined reference to `pthread_mutex_lock' 97</pre> 98</blockquote> 99 100<p> Add the "-lpthread" library to the "make makefiles" command. </p> 101 102<blockquote> 103<pre> 104% make makefiles .... AUXLIBS_LMDB="... -lpthread" 105</pre> 106</blockquote> 107 108<h2><a name="configure">Configuring LMDB settings</a></h2> 109 110<p> Postfix provides one configuration parameter that controls 111LMDB database behavior. </p> 112 113<ul> 114 115<li> <p> lmdb_map_size (default: 16777216). This setting specifies 116the initial LMDB database size limit in bytes. Each time a database 117becomes "full", its size limit is doubled. The maximum size is the 118largest signed integer value of "long". </p> 119 120</ul> 121 122<h2> <a name="locking">Using LMDB maps with non-Postfix programs</a> </h2> 123 124<p> Programs that use LMDB's built-in locking protocol will corrupt 125a Postfix LMDB database or will read garbage. </p> 126 127<p> Postfix does not use LMDB's built-in locking protocol, because 128that would require world-writable lockfiles, and would violate 129Postfix security policy. Instead, Postfix uses external locks based 130on fcntl(2) to prevent writers from corrupting the database, and 131to prevent readers from receiving garbage. </p> 132 133<p> See lmdb_table(5) for a detailed description of the locking 134protocol that all programs must use when they access a Postfix LMDB 135database. </p> 136 137<h2> <a name="supported"> Required minimum LMDB patchlevel </a> </h2> 138 139<p> Currently, Postfix requires LMDB 0.9.11 or later. The required 140minimum LMDB patchlevel has evolved over time, as the result of 141Postfix deployment experience: </p> 142 143<ul> 144 145<li> <p> LMDB 0.9.11 allows Postfix daemons to log an LMDB error 146message, instead of falling out of the sky without any notification. 147</p> 148 149<li> <p> LMDB 0.9.10 closes an information leak where LMDB was 150writing up to 4-kbyte chunks of uninitialized heap memory to the 151database. This would persist information that was not meant to be 152persisted, or share information that was not meant to be shared. 153</p> 154 155<li> <p> LMDB 0.9.9 allows Postfix to use external (fcntl()-based) 156locks, instead of having to use world-writable LMDB lock files, 157violating the Postfix security model in multiple ways. </p> 158 159<li> <p> LMDB 0.9.8 allows Postfix to recover from a "database full" 160error without having to close the database. This version adds support 161to update the database size limit on-the-fly. This is necessary 162because Postfix database sizes vary with mail server load. </p> 163 164<li> <p> LMDB 0.9.7 allows the postmap(1) and postalias(1) commands 165to use a bulk-mode transaction larger than the amount of physical 166memory. This is necessary because LMDB supports databases larger 167than physical memory. </p> 168 169</ul> 170 171<h2> <a name="credits"> Credits</a> </h2> 172 173<ul> 174 175<li> <p> Howard Chu contributed the initial Postfix dict_lmdb driver. 176</p> 177 178<li> <p> Wietse Venema wrote an abstraction layer (slmdb) that 179behaves more like Berkeley DB, NDBM, etc. This layer automatically 180retries an LMDB request when a database needs to be resized, or 181after a database was resized by a different process. </p> 182 183<li> <p> Howard and Wietse went through many iterations with changes 184to both LMDB and Postfix, with input from Viktor Dukhovni. </p> 185 186</ul> 187 188<!-- 189 190<h2><a name="limitations">Unexpected failure modes of Postfix LMDB 191databases. </a> </h2> 192 193<p> As documented below, conversion to LMDB introduces a number of 194failure modes that don't exist with other Postfix databases. Some 195failure modes have been eliminated in the course of time. 196The writeup below reflects the status as of LMDB 0.9.9. </p> 197 198--> 199 200<!-- 201 202<p> <strong>Unexpected "Permission denied" errors. </strong></p> 203 204<dl> 205 206<dt> Problem: </dt> <dd> <p> A world-readable LMDB database cannot 207be opened by a process with a UID that differs from the database 208file owner, even when an attempt is made to open the database 209read-only. This problem does not exist with other Postfix databases. 210</p> </dd> 211 212<dt> Background: </dt> <dd> <p> The LMDB implementation requires 213write access to maintain read locks, and perhaps for other purposes. 214</p> </dd> 215 216<dt> Solution: </dt> <dd> <p> Consider using cdb: to manage root-owned 217databases under the root-owned <tt>/etc</tt> or config_directory 218(default: <tt>/etc/postfix</tt>) such as access(5), virtual(5), 219transport(5). Support to create LMDB databases is available only 220for unprivileged Postfix daemon processes such as postscreen(8), 221tlsmgr(8) and verify(8) that manage postfix-owned databases under 222the postfix-owned data_directory (default: <tt>/var/lib/postfix</tt>). 223</p> </dd> 224 225</dl> 226 227--> 228 229<!-- 230 231<p> <strong>Unexpected "readers full" errors. </strong></p> 232 233<dl> 234 235<dt> Problem: </dt> <dd> <p> Under heavy load, database read 236operations fail with MDB_READERS_FULL errors. This problem does not 237exist with other Postfix databases. </p> </dd> 238 239<dt> Background: </dt> <dd> <p> The LMDB implementation enforces a 240hard limit on the number of simultaneous read requests for the same 241database environment. This limit must be specified in advance with 242the lmdb_max_readers configuration parameter. </p> </dd> 243 244<dt> Mitigation: </dt> <dd> <p> Postfix logs a warning suggesting 245that the lmdb_max_readers parameter value be increased, and retries 246the failed operation for a limited number of times while running 247with reduced performance. </p> </dd> 248 249<dt> Prevention: </dt> <dd> <p> Monitor your LMDB files for 250MDB_READERS_FULL errors. After making the necessary adjustments, 251restart Postfix. </p> </dd> 252 253</dl> 254 255--> 256 257<!-- 258 259<p> <strong>Unexpected postmap(1)/postalias(1) "database full" 260errors. </strong></p> 261 262<dl> 263 264<dt> Problem: </dt> <dd> <p> The "postmap lmdb:filename" command 265fails with an MDB_TXN_FULL error. This problem does not exist with 266other Postfix databases. </p> </dd> 267 268<dt> Background: </dt> 269 270<dd> 271 272<p> The LMDB implementation has a hard limit on the total transaction 273size. This limit is independent of the LMDB database size. Therefore, 274the problem cannot be resolved by increasing the lmdb_map_size 275value. </p> 276 277<p> This symptom is indicative of a flawed design. All LMDB data 278structures should share the same storage pool so that they can scale 279with the database size, and so that all "out of storage" errors are 280resolved by increasing the database size. </p> </dd> 281 282--> 283 284<!-- 285 286<p> Problem: </dt> <dd> <p> The "postmap lmdb:filename" command 287fails with an MDB_MAP_FULL error. This problem does not exist with 288other Postfix databases. </p> </dd> 289 290<dl> 291 292<dt> Background: </dt> 293 294<dd> 295 296<p> LMDB databases have a hard size limit (configured with the 297lmdb_map_size configuration parameter). </p> 298 299<p> When executing "postmap lmdb:filename", the Postfix LMDB database 300client stores the new data in a transaction which takes up space 301in addition to the existing data, and commits the transaction when 302it closes the database. Only then can the space for old data be 303reused. </p> 304 305</dd> 306 307<dt> Impact: </dt> <dd> <p> This failure does not affect Postfix 308availability, because the old data still exists in the database. 309</p> </dd> 310 311<dt> Mitigation: </dt> <dd> 312 313<p> When the postmap(1) or postalias(1) command fails with an 314MDB_MAP_FULL error, it expands the database file size to the current 315LMDB map size limit before terminating. </p> 316 317<p> Next, when you re-run the postmap(1) or postalias(1) command, 318it discovers that the LMDB file is larger than lmdb_map_size/3, 319logs a warning, and uses a larger LMDB map size limit instead: </p> 320 321<p> <tt> warning: <i>filename</i>.lmdb: file size 15024128 ≥ 322(lmdb map size limit 16777216)/3<br> warning: <i>filename</i>.lmdb: 323using map size limit 45072384</tt> </p> 324 325<p> By repeating the two steps above you can automate recovery and 326avoid the need for human intervention. Just repeat "postmap 327lmdb:filename" (up to some limit). After each failure it will use 328a 3x larger size limit, and eventually the "database full" error 329should disappear. This fails only when the disk is full or when 330the LMDB map size limit would exceed the memory address space size 331limit. </p> 332 333<dt> Prevention: </dt> <dd> <p> Monitor your LMDB files and make 334sure that in main.cf, lmdb_map_size > 3x the largest LMDB file 335size. </p> </dd> </dl> 336 337</dl> 338 339--> 340 341<!-- 342 343<p> <strong>Unexpected Postfix daemon "database full" errors. 344</strong></p> 345 346<dl> 347 348<dt> Problem: </dt> <dd> <p> Postfix daemon programs fail with 349"database full" errors, such as postscreen(8), tlsmgr(8) or verify(8). 350This problem does not exist with other Postfix databases. </p> 351</dd> 352 353<dt> Impact: </dt> <dd> <p> This failure temporarily affects Postfix 354availability. The daemon restarts automatically and tries to open 355the database again as described next. </p> </dd> 356 357<dt> Mitigation: </dt> <dd> <p> When a Postfix daemon opens an LMDB 358file larger than lmdb_map_size/3, it logs a warning and uses a 359larger size limit instead: </p> 360 361<p> <tt> warning: <i>filename</i>.lmdb: file size 15024128 ≥ 362(lmdb map size limit 16777216)/3 <br>warning: <i>filename</i>.lmdb: 363using map size limit 45072384</tt> </p> 364 365<p> This can be used to automate recovery and avoid the need for 366human intervention. Each time the daemon runs into a "database full" 367error, it restarts and uses a 3x larger size limit. The "database 368full" error will disappear, at least for a while. </p> 369 370<dt> Prevention: </dt> <dd> <p> Monitor your LMDB files and make 371sure that lmdb_map_size > 3x the largest LMDB file size. </p> 372</dd> </dl> 373 374--> 375 376<!-- 377 378<p> <strong>Non-obvious recovery with postmap(1), postalias(1), or 379tlsmgr(8) from a corrupted database. </strong></p> 380 381<dl> 382 383<dt> Problem: </dt> <dd> <p> A corrupted LMDB database can't be 384rebuilt simply by re-running postmap(1) or postalias(1), or by 385waiting until a tlsmgr(8) daemon restarts. This problem does not 386exist with other Postfix databases. </p> </dd> 387 388<dt> Background: </dt> <dd> <p> The Postfix LMDB database client 389does not truncate the database file. Instead it attempts to create 390a transaction for a "drop" request plus subsequent "store" requests. 391That is obviously not possible with a corrupted database file. </p> 392</dd> 393 394<dt> Impact: </dt> <dd> <p> Postfix does not process mail until 395someone fixes the problem. </p> </dd> 396 397<dt> Recovery: </dt> <dd> <p> First delete the ".lmdb" file by hand. 398Then rebuild the file with the postmap(1) or postalias(1) 399command if the file was created with those commands, or restart 400postfix daemons if the file is maintained by tlsmgr(8). 401</p> </dd> 402 403<dt> Prevention: </dt> <dd> 404 405<p> Arrange your file systems such that they never run out of free 406space. </p> 407 408<p> Use ECC memory to detect and correct silent corruption of 409in-memory file system data and metadata. </p> 410 411<p> Use a file system such as ZFS to detect and correct silent 412corruption of on-disk file system data and metadata. DO NOT 413use ZFS on systems without ECC memory error correction. </p> 414 415</dd> </dl> 416 417--> 418 419</body> 420 421</html> 422