1# $OpenLDAP$ 2# Copyright 1999-2017 The OpenLDAP Foundation, All Rights Reserved. 3# COPYING RESTRICTIONS APPLY, see COPYRIGHT. 4 5H1: Replication 6 7Replicated directories are a fundamental requirement for delivering a 8resilient enterprise deployment. 9 10{{PRD:OpenLDAP}} has various configuration options for creating a replicated 11directory. In previous releases, replication was discussed in terms of 12a {{master}} server and some number of {{slave}} servers. A master 13accepted directory updates from other clients, and a slave only 14accepted updates from a (single) master. The replication structure 15was rigidly defined and any particular database could only fulfill 16a single role, either master or slave. 17 18As OpenLDAP now supports a wide variety of replication topologies, these 19terms have been deprecated in favor of {{provider}} and 20{{consumer}}: A provider replicates directory updates to consumers; 21consumers receive replication updates from providers. Unlike the 22rigidly defined master/slave relationships, provider/consumer roles 23are quite fluid: replication updates received in a consumer can be 24further propagated by that consumer to other servers, so a consumer 25can also act simultaneously as a provider. Also, a consumer need not 26be an actual LDAP server; it may be just an LDAP client. 27 28The following sections will describe the replication technology and 29discuss the various replication options that are available. 30 31H2: Replication Technology 32 33H3: LDAP Sync Replication 34 35The {{TERM:LDAP Sync}} Replication engine, {{TERM:syncrepl}} for 36short, is a consumer-side replication engine that enables the 37consumer {{TERM:LDAP}} server to maintain a shadow copy of a 38{{TERM:DIT}} fragment. A syncrepl engine resides at the consumer 39and executes as one of the {{slapd}}(8) threads. It creates and maintains a 40consumer replica by connecting to the replication provider to perform 41the initial DIT content load followed either by periodic content 42polling or by timely updates upon content changes. 43 44Syncrepl uses the LDAP Content Synchronization protocol (or LDAP Sync for 45short) as the replica synchronization protocol. LDAP Sync provides 46a stateful replication which supports both pull-based and push-based 47synchronization and does not mandate the use of a history store. 48In pull-based replication the consumer periodically 49polls the provider for updates. In push-based replication the consumer 50listens for updates that are sent by the provider in realtime. Since the 51protocol does not require a history store, the provider does not need to 52maintain any log of updates it has received (Note 53that the syncrepl engine is extensible and additional replication 54protocols may be supported in the future.). 55 56Syncrepl keeps track of the status of the replication content by 57maintaining and exchanging synchronization cookies. Because the 58syncrepl consumer and provider maintain their content status, the 59consumer can poll the provider content to perform incremental 60synchronization by asking for the entries required to make the 61consumer replica up-to-date with the provider content. Syncrepl 62also enables convenient management of replicas by maintaining replica 63status. The consumer replica can be constructed from a consumer-side 64or a provider-side backup at any synchronization status. Syncrepl 65can automatically resynchronize the consumer replica up-to-date 66with the current provider content. 67 68Syncrepl supports both pull-based and push-based synchronization. 69In its basic refreshOnly synchronization mode, the provider uses 70pull-based synchronization where the consumer servers need not be 71tracked and no history information is maintained. The information 72required for the provider to process periodic polling requests is 73contained in the synchronization cookie of the request itself. To 74optimize the pull-based synchronization, syncrepl utilizes the 75present phase of the LDAP Sync protocol as well as its delete phase, 76instead of falling back on frequent full reloads. To further optimize 77the pull-based synchronization, the provider can maintain a per-scope 78session log as a history store. In its refreshAndPersist mode of 79synchronization, the provider uses a push-based synchronization. 80The provider keeps track of the consumer servers that have requested 81a persistent search and sends them necessary updates as the provider 82replication content gets modified. 83 84With syncrepl, a consumer server can create a replica without 85changing the provider's configurations and without restarting the 86provider server, if the consumer server has appropriate access 87privileges for the DIT fragment to be replicated. The consumer 88server can stop the replication also without the need for provider-side 89changes and restart. 90 91Syncrepl supports partial, sparse, and fractional replications. The shadow 92DIT fragment is defined by a general search criteria consisting of 93base, scope, filter, and attribute list. The replica content is 94also subject to the access privileges of the bind identity of the 95syncrepl replication connection. 96 97 98H4: The LDAP Content Synchronization Protocol 99 100The LDAP Sync protocol allows a client to maintain a synchronized 101copy of a DIT fragment. The LDAP Sync operation is defined as a set 102of controls and other protocol elements which extend the LDAP search 103operation. This section introduces the LDAP Content Sync protocol 104only briefly. For more information, refer to {{REF:RFC4533}}. 105 106The LDAP Sync protocol supports both polling and listening for changes 107by defining two respective synchronization operations: 108{{refreshOnly}} and {{refreshAndPersist}}. Polling is implemented 109by the {{refreshOnly}} operation. The consumer 110polls the provider using an LDAP Search request with an LDAP Sync 111control attached. The consumer copy is synchronized 112to the provider copy at the time of polling using the information 113returned in the search. The provider finishes the 114search operation by returning {{SearchResultDone}} at the end of 115the search operation as in the normal search. Listening is 116implemented by the {{refreshAndPersist}} operation. As the name 117implies, it begins with a search, like refreshOnly. Instead of 118finishing the search after returning all entries currently matching 119the search criteria, the synchronization search remains persistent 120in the provider. Subsequent updates to the synchronization content 121in the provider cause additional entry updates to be sent to the 122consumer. 123 124The {{refreshOnly}} operation and the refresh stage of the 125{{refreshAndPersist}} operation can be performed with a present 126phase or a delete phase. 127 128In the present phase, the provider sends the consumer the entries updated 129within the search scope since the last synchronization. The provider 130sends all requested attributes, be they changed or not, of the updated 131entries. For each unchanged entry which remains in the scope, the 132provider sends a present message consisting only of the name of the 133entry and the synchronization control representing state present. 134The present message does not contain any attributes of the entry. 135After the consumer receives all update and present entries, it can 136reliably determine the new consumer copy by adding the entries added 137to the provider, by replacing the entries modified at the provider, and 138by deleting entries in the consumer copy which have not been updated 139nor specified as being present at the provider. 140 141The transmission of the updated entries in the delete phase is the 142same as in the present phase. The provider sends all the requested 143attributes of the entries updated within the search scope since the 144last synchronization to the consumer. In the delete phase, however, 145the provider sends a delete message for each entry deleted from the 146search scope, instead of sending present messages. The delete 147message consists only of the name of the entry and the synchronization 148control representing state delete. The new consumer copy can be 149determined by adding, modifying, and removing entries according to 150the synchronization control attached to the {{SearchResultEntry}} 151message. 152 153In the case that the LDAP Sync provider maintains a history store and 154can determine which entries are scoped out of the consumer copy since 155the last synchronization time, the provider can use the delete phase. 156If the provider does not maintain any history store, cannot determine 157the scoped-out entries from the history store, or the history store 158does not cover the outdated synchronization state of the consumer, 159the provider should use the present phase. The use of the present 160phase is much more efficient than a full content reload in terms 161of the synchronization traffic. To reduce the synchronization 162traffic further, the LDAP Sync protocol also provides several 163optimizations such as the transmission of the normalized {{EX:entryUUID}}s 164and the transmission of multiple {{EX:entryUUIDs}} in a single 165{{syncIdSet}} message. 166 167At the end of the {{refreshOnly}} synchronization, the provider sends 168a synchronization cookie to the consumer as a state indicator of the 169consumer copy after the synchronization is completed. The consumer 170will present the received cookie when it requests the next incremental 171synchronization to the provider. 172 173When {{refreshAndPersist}} synchronization is used, the provider sends 174a synchronization cookie at the end of the refresh stage by sending 175a Sync Info message with refreshDone=TRUE. It also sends a 176synchronization cookie by attaching it to {{SearchResultEntry}} 177messages generated in the persist stage of the synchronization search. During 178the persist stage, the provider can also send a Sync Info message 179containing the synchronization cookie at any time the provider wants 180to update the consumer-side state indicator. 181 182In the LDAP Sync protocol, entries are uniquely identified by the 183{{EX:entryUUID}} attribute value. It can function as a reliable 184identifier of the entry. The DN of the entry, on the other hand, 185can be changed over time and hence cannot be considered as the 186reliable identifier. The {{EX:entryUUID}} is attached to each 187{{SearchResultEntry}} or {{SearchResultReference}} as a part of the 188synchronization control. 189 190H4: Syncrepl Details 191 192The syncrepl engine utilizes both the {{refreshOnly}} and the 193{{refreshAndPersist}} operations of the LDAP Sync protocol. If a 194syncrepl specification is included in a database definition, 195{{slapd}}(8) launches a syncrepl engine as a {{slapd}}(8) thread 196and schedules its execution. If the {{refreshOnly}} operation is 197specified, the syncrepl engine will be rescheduled at the interval 198time after a synchronization operation is completed. If the 199{{refreshAndPersist}} operation is specified, the engine will remain 200active and process the persistent synchronization messages from the 201provider. 202 203The syncrepl engine utilizes both the present phase and the delete 204phase of the refresh synchronization. It is possible to configure 205a session log in the provider which stores the 206{{EX:entryUUID}}s of a finite number of entries deleted from a 207database. Multiple replicas share the same session log. The syncrepl 208engine uses the 209delete phase if the session log is present and the state of the 210consumer server is recent enough that no session log entries are 211truncated after the last synchronization of the client. The syncrepl 212engine uses the present phase if no session log is configured for 213the replication content or if the consumer replica is too outdated 214to be covered by the session log. The current design of the session 215log store is memory based, so the information contained in the 216session log is not persistent over multiple provider invocations. 217It is not currently supported to access the session log store by 218using LDAP operations. It is also not currently supported to impose 219access control to the session log. 220 221As a further optimization, even in the case the synchronization 222search is not associated with any session log, no entries will be 223transmitted to the consumer server when there has been no update 224in the replication context. 225 226The syncrepl engine, which is a consumer-side replication engine, 227can work with any backends. The LDAP Sync provider can be configured 228as an overlay on any backend, but works best with the {{back-bdb}}, 229{{back-hdb}}, or {{back-mdb}} backends. 230 231The LDAP Sync provider maintains a {{EX:contextCSN}} for each 232database as the current synchronization state indicator of the 233provider content. It is the largest {{EX:entryCSN}} in the provider 234context such that no transactions for an entry having smaller 235{{EX:entryCSN}} value remains outstanding. The {{EX:contextCSN}} 236could not just be set to the largest issued {{EX:entryCSN}} because 237{{EX:entryCSN}} is obtained before a transaction starts and 238transactions are not committed in the issue order. 239 240The provider stores the {{EX:contextCSN}} of a context in the 241{{EX:contextCSN}} attribute of the context suffix entry. The attribute 242is not written to the database after every update operation though; 243instead it is maintained primarily in memory. At database start 244time the provider reads the last saved {{EX:contextCSN}} into memory 245and uses the in-memory copy exclusively thereafter. By default, 246changes to the {{EX:contextCSN}} as a result of database updates 247will not be written to the database until the server is cleanly 248shut down. A checkpoint facility exists to cause the {{EX:contextCSN}} to 249be written out more frequently if desired. 250 251Note that at startup time, if the provider is unable to read a 252{{EX:contextCSN}} from the suffix entry, it will scan the entire 253database to determine the value, and this scan may take quite a 254long time on a large database. When a {{EX:contextCSN}} value is 255read, the database will still be scanned for any {{EX:entryCSN}} 256values greater than it, to make sure the {{EX:contextCSN}} value 257truly reflects the greatest committed {{EX:entryCSN}} in the database. 258On databases which support inequality indexing, setting an eq index 259on the {{EX:entryCSN}} attribute and configuring {{contextCSN}} 260checkpoints will greatly speed up this scanning step. 261 262If no {{EX:contextCSN}} can be determined by reading and scanning 263the database, a new value will be generated. Also, if scanning the 264database yielded a greater {{EX:entryCSN}} than was previously 265recorded in the suffix entry's {{EX:contextCSN}} attribute, a 266checkpoint will be immediately written with the new value. 267 268The consumer also stores its replica state, which is the provider's 269{{EX:contextCSN}} received as a synchronization cookie, in the 270{{EX:contextCSN}} attribute of the suffix entry. The replica state 271maintained by a consumer server is used as the synchronization state 272indicator when it performs subsequent incremental synchronization 273with the provider server. It is also used as a provider-side 274synchronization state indicator when it functions as a secondary 275provider server in a cascading replication configuration. Since 276the consumer and provider state information are maintained in the 277same location within their respective databases, any consumer can 278be promoted to a provider (and vice versa) without any special 279actions. 280 281Because a general search filter can be used in the syncrepl 282specification, some entries in the context may be omitted from the 283synchronization content. The syncrepl engine creates a glue entry 284to fill in the holes in the replica context if any part of the 285replica content is subordinate to the holes. The glue entries will 286not be returned in the search result unless {{ManageDsaIT}} control 287is provided. 288 289Also as a consequence of the search filter used in the syncrepl 290specification, it is possible for a modification to remove an entry 291from the replication scope even though the entry has not been deleted 292on the provider. Logically the entry must be deleted on the consumer 293but in {{refreshOnly}} mode the provider cannot detect and propagate 294this change without the use of the session log on the provider. 295 296For configuration, please see the {{SECT:Syncrepl}} section. 297 298 299H2: Deployment Alternatives 300 301While the LDAP Sync specification only defines a narrow scope for replication, 302the OpenLDAP implementation is extremely flexible and supports a variety of 303operating modes to handle other scenarios not explicitly addressed in the spec. 304 305 306H3: Delta-syncrepl replication 307 308* Disadvantages of LDAP Sync replication: 309 310LDAP Sync replication is an object-based replication mechanism. 311When any attribute value in a replicated object is changed on the provider, 312each consumer fetches and processes the complete changed object, including 313{{B:both the changed and unchanged attribute values}} during replication. 314One advantage of this approach is that when multiple changes occur to 315a single object, the precise sequence of those changes need not be preserved; 316only the final state of the entry is significant. But this approach 317may have drawbacks when the usage pattern involves single changes to 318multiple objects. 319 320For example, suppose you have a database consisting of 102,400 objects of 1 KB 321each. Further, suppose you routinely run a batch job to change the value of 322a single two-byte attribute value that appears in each of the 102,400 objects 323on the master. Not counting LDAP and TCP/IP protocol overhead, each time you 324run this job each consumer will transfer and process {{B:100 MB}} of data to 325process {{B:200KB of changes!}} 326 32799.98% of the data that is transmitted and processed in a case like this will 328be redundant, since it represents values that did not change. This is a waste 329of valuable transmission and processing bandwidth and can cause an unacceptable 330replication backlog to develop. While this situation is extreme, it serves to 331demonstrate a very real problem that is encountered in some LDAP deployments. 332 333 334* Where Delta-syncrepl comes in: 335 336Delta-syncrepl, a changelog-based variant of syncrepl, is designed to address 337situations like the one described above. Delta-syncrepl works by maintaining a 338changelog of a selectable depth in a separate database on the provider. The replication consumer 339checks the changelog for the changes it needs and, as long as 340the changelog contains the needed changes, the consumer fetches the changes 341from the changelog and applies them to its database. If, however, a replica 342is too far out of sync (or completely empty), conventional syncrepl is used to 343bring it up to date and replication then switches back to the delta-syncrepl 344mode. 345 346Note: since the database state is stored in both the changelog DB and the 347main DB on the provider, it is important to backup/restore both the changelog 348DB and the main DB using slapcat/slapadd when restoring a DB or copying 349it to another machine. 350 351For configuration, please see the {{SECT:Delta-syncrepl}} section. 352 353 354H3: N-Way Multi-Master replication 355 356Multi-Master replication is a replication technique using Syncrepl to replicate 357data to multiple provider ("Master") Directory servers. 358 359H4: Valid Arguments for Multi-Master replication 360 361* If any provider fails, other providers will continue to accept updates 362* Avoids a single point of failure 363* Providers can be located in several physical sites i.e. distributed across 364the network/globe. 365* Good for Automatic failover/High Availability 366 367H4: Invalid Arguments for Multi-Master replication 368 369(These are often claimed to be advantages of Multi-Master replication but 370those claims are false): 371 372* It has {{B:NOTHING}} to do with load balancing 373* Providers {{B:must}} propagate writes to {{B:all}} the other servers, which 374means the network traffic and write load spreads across all 375of the servers the same as for single-master. 376* Server utilization and performance are at best identical for 377Multi-Master and Single-Master replication; at worst Single-Master is 378superior because indexing can be tuned differently to optimize for the 379different usage patterns between the provider and the consumers. 380 381H4: Arguments against Multi-Master replication 382 383* Breaks the data consistency guarantees of the directory model 384* {{URL:http://www.openldap.org/faq/data/cache/1240.html}} 385* If connectivity with a provider is lost because of a network partition, then 386"automatic failover" can just compound the problem 387* Typically, a particular machine cannot distinguish between losing contact 388 with a peer because that peer crashed, or because the network link has failed 389* If a network is partitioned and multiple clients start writing to each of the 390"masters" then reconciliation will be a pain; it may be best to simply deny 391writes to the clients that are partitioned from the single provider 392 393 394For configuration, please see the {{SECT:N-Way Multi-Master}} section below 395 396H3: MirrorMode replication 397 398MirrorMode is a hybrid configuration that provides all of the consistency 399guarantees of single-master replication, while also providing the high 400availability of multi-master. In MirrorMode two providers are set up to 401replicate from each other (as a multi-master configuration), but an 402external frontend is employed to direct all writes to only one of 403the two servers. The second provider will only be used for writes if 404the first provider crashes, at which point the frontend will switch to 405directing all writes to the second provider. When a crashed provider is 406repaired and restarted it will automatically catch up to any changes 407on the running provider and resync. 408 409H4: Arguments for MirrorMode 410 411* Provides a high-availability (HA) solution for directory writes (replicas handle reads) 412* As long as one provider is operational, writes can safely be accepted 413* Provider nodes replicate from each other, so they are always up to date and 414can be ready to take over (hot standby) 415* Syncrepl also allows the provider nodes to re-synchronize after any downtime 416 417 418H4: Arguments against MirrorMode 419 420* MirrorMode is not what is termed as a Multi-Master solution. This is because 421writes have to go to just one of the mirror nodes at a time 422* MirrorMode can be termed as Active-Active Hot-Standby, therefore an external 423server (slapd in proxy mode) or device (hardware load balancer) 424is needed to manage which provider is currently active 425* Backups are managed slightly differently 426- If backing up the Berkeley database itself and periodically backing up the 427transaction log files, then the same member of the mirror pair needs to be 428used to collect logfiles until the next database backup is taken 429 430For configuration, please see the {{SECT:MirrorMode}} section below 431 432 433H3: Syncrepl Proxy Mode 434 435While the LDAP Sync protocol supports both pull- and push-based replication, 436the push mode (refreshAndPersist) must still be initiated from the consumer 437before the provider can begin pushing changes. In some network configurations, 438particularly where firewalls restrict the direction in which connections 439can be made, a provider-initiated push mode may be needed. 440 441This mode can be configured with the aid of the LDAP Backend 442({{SECT: Backends}} and {{slapd-ldap(8)}}). Instead of running the 443syncrepl engine on the actual consumer, a slapd-ldap proxy is set up 444near (or collocated with) the provider that points to the consumer, 445and the syncrepl engine runs on the proxy. 446 447For configuration, please see the {{SECT:Syncrepl Proxy}} section. 448 449H4: Replacing Slurpd 450 451The old {{slurpd}} mechanism only operated in provider-initiated 452push mode. Slurpd replication was deprecated in favor of Syncrepl 453replication and has been completely removed from OpenLDAP 2.4. 454 455The slurpd daemon was the original replication mechanism inherited from 456UMich's LDAP and operated in push mode: the master pushed changes to the 457slaves. It was replaced for many reasons, in brief: 458 459 * It was not reliable 460 ** It was extremely sensitive to the ordering of records in the replog 461 ** It could easily go out of sync, at which point manual intervention was 462 required to resync the slave database with the master directory 463 ** It wasn't very tolerant of unavailable servers. If a slave went down 464 for a long time, the replog could grow to a size that was too large for 465 slurpd to process 466 * It only worked in push mode 467 * It required stopping and restarting the master to add new slaves 468 * It only supported single master replication 469 470Syncrepl has none of those weaknesses: 471 472 * Syncrepl is self-synchronizing; you can start with a consumer database 473 in any state from totally empty to fully synced and it will automatically 474 do the right thing to achieve and maintain synchronization 475 ** It is completely insensitive to the order in which changes occur 476 ** It guarantees convergence between the consumer and the provider 477 content without manual intervention 478 ** It can resynchronize regardless of how long a consumer stays out 479 of contact with the provider 480 * Syncrepl can operate in either direction 481 * Consumers can be added at any time without touching anything on the 482 provider 483 * Multi-master replication is supported 484 485 486H2: Configuring the different replication types 487 488H3: Syncrepl 489 490H4: Syncrepl configuration 491 492Because syncrepl is a consumer-side replication engine, the syncrepl 493specification is defined in {{slapd.conf}}(5) of the consumer 494server, not in the provider server's configuration file. The initial 495loading of the replica content can be performed either by starting 496the syncrepl engine with no synchronization cookie or by populating 497the consumer replica by loading an {{TERM:LDIF}} file dumped as a 498backup at the provider. 499 500When loading from a backup, it is not required to perform the initial 501loading from the up-to-date backup of the provider content. The 502syncrepl engine will automatically synchronize the initial consumer 503replica to the current provider content. As a result, it is not 504required to stop the provider server in order to avoid the replica 505inconsistency caused by the updates to the provider content during 506the content backup and loading process. 507 508When replicating a large scale directory, especially in a bandwidth 509constrained environment, it is advised to load the consumer replica 510from a backup instead of performing a full initial load using 511syncrepl. 512 513 514H4: Set up the provider slapd 515 516The provider is implemented as an overlay, so the overlay itself 517must first be configured in {{slapd.conf}}(5) before it can be 518used. The provider has two primary configuration directives and 519two secondary directives for when delta-syncrepl is being used. 520Because the LDAP Sync search is subject to access control, proper 521access control privileges should be set up for the replicated 522content. 523 524The two primary options to configure are the checkpoint and 525sessionlog behaviors. 526 527The {{EX:contextCSN}} checkpoint is configured by the 528 529> syncprov-checkpoint <ops> <minutes> 530 531directive. Checkpoints are only tested after successful write 532operations. If {{<ops>}} operations or more than {{<minutes>}} 533time has passed since the last checkpoint, a new checkpoint is 534performed. Checkpointing is disabled by default. 535 536The session log is configured by the 537 538> syncprov-sessionlog <ops> 539 540directive, where {{<ops>}} is the maximum number of session log 541entries the session log can record. All write operations (except Adds) 542are recorded in the log. 543 544Note that using the session log requires searching on the {{entryUUID}} 545attribute. Setting an eq index on this attribute will greatly benefit 546the performance of the session log on the provider. 547 548The reloadhint option is configured by the 549 550> syncprov-reloadhint <TRUE|FALSE> 551 552directive. It must be set TRUE when using the accesslog overlay for 553delta-based syncrepl replication support. The default is FALSE. 554 555The nonpresent option should only be configured if the overlay is 556being placed on top of a log database, such as when used with 557delta-syncrepl. 558 559The nonpresent option is configured by the 560 561> syncprov-nopresent <TRUE|FALSE> 562 563directive. This value should only be set TRUE for a syncprov instance 564on top of a log database (such as one managed by the accesslog overlay). 565The default is FALSE. 566 567A more complete example of the {{slapd.conf}}(5) content is thus: 568 569> database mdb 570> maxsize 1073741824 571> suffix dc=Example,dc=com 572> rootdn dc=Example,dc=com 573> directory /var/ldap/db 574> index objectclass,entryCSN,entryUUID eq 575> 576> overlay syncprov 577> syncprov-checkpoint 100 10 578> syncprov-sessionlog 100 579 580 581H4: Set up the consumer slapd 582 583The syncrepl replication is specified in the database section of 584{{slapd.conf}}(5) for the replica context. The syncrepl engine 585is backend independent and the directive can be defined with any 586database type. 587 588> database mdb 589> maxsize 1073741824 590> suffix dc=Example,dc=com 591> rootdn dc=Example,dc=com 592> directory /var/ldap/db 593> index objectclass,entryCSN,entryUUID eq 594> 595> syncrepl rid=123 596> provider=ldap://provider.example.com:389 597> type=refreshOnly 598> interval=01:00:00:00 599> searchbase="dc=example,dc=com" 600> filter="(objectClass=organizationalPerson)" 601> scope=sub 602> attrs="cn,sn,ou,telephoneNumber,title,l" 603> schemachecking=off 604> bindmethod=simple 605> binddn="cn=syncuser,dc=example,dc=com" 606> credentials=secret 607 608In this example, the consumer will connect to the provider {{slapd}}(8) 609at port 389 of {{FILE:ldap://provider.example.com}} to perform a 610polling ({{refreshOnly}}) mode of synchronization once a day. It 611will bind as {{EX:cn=syncuser,dc=example,dc=com}} using simple 612authentication with password "secret". Note that the access control 613privilege of {{EX:cn=syncuser,dc=example,dc=com}} should be set 614appropriately in the provider to retrieve the desired replication 615content. Also the search limits must be high enough on the provider 616to allow the syncuser to retrieve a complete copy of the requested 617content. The consumer uses the rootdn to write to its database so 618it always has full permissions to write all content. 619 620The synchronization search in the above example will search for the 621entries whose objectClass is organizationalPerson in the entire 622subtree rooted at {{EX:dc=example,dc=com}}. The requested attributes 623are {{EX:cn}}, {{EX:sn}}, {{EX:ou}}, {{EX:telephoneNumber}}, 624{{EX:title}}, and {{EX:l}}. The schema checking is turned off, so 625that the consumer {{slapd}}(8) will not enforce entry schema 626checking when it processes updates from the provider {{slapd}}(8). 627 628For more detailed information on the syncrepl directive, see the 629{{SECT:syncrepl}} section of {{SECT:The slapd Configuration File}} 630chapter of this admin guide. 631 632 633H4: Start the provider and the consumer slapd 634 635The provider {{slapd}}(8) is not required to be restarted. 636{{contextCSN}} is automatically generated as needed: it might be 637originally contained in the {{TERM:LDIF}} file, generated by 638{{slapadd}} (8), generated upon changes in the context, or generated 639when the first LDAP Sync search arrives at the provider. If an 640LDIF file is being loaded which did not previously contain the 641{{contextCSN}}, the {{-w}} option should be used with {{slapadd}} 642(8) to cause it to be generated. This will allow the server to 643startup a little quicker the first time it runs. 644 645When starting a consumer {{slapd}}(8), it is possible to provide 646a synchronization cookie as the {{-c cookie}} command line option 647in order to start the synchronization from a specific state. The 648cookie is a comma separated list of name=value pairs. Currently 649supported syncrepl cookie fields are {{csn=<csn>}} and {{rid=<rid>}}. 650{{<csn>}} represents the current synchronization state of the 651consumer replica. {{<rid>}} identifies a consumer replica locally 652within the consumer server. It is used to relate the cookie to the 653syncrepl definition in {{slapd.conf}}(5) which has the matching 654replica identifier. The {{<rid>}} must have no more than 3 decimal 655digits. The command line cookie overrides the synchronization 656cookie stored in the consumer replica database. 657 658 659H3: Delta-syncrepl 660 661H4: Delta-syncrepl Provider configuration 662 663Setting up delta-syncrepl requires configuration changes on both the master and 664replica servers: 665 666> # Give the replica DN unlimited read access. This ACL needs to be 667> # merged with other ACL statements, and/or moved within the scope 668> # of a database. The "by * break" portion causes evaluation of 669> # subsequent rules. See slapd.access(5) for details. 670> access to * 671> by dn.base="cn=replicator,dc=symas,dc=com" read 672> by * break 673> 674> # Set the module path location 675> modulepath /opt/symas/lib/openldap 676> 677> # Load the hdb backend 678> moduleload back_hdb.la 679> 680> # Load the accesslog overlay 681> moduleload accesslog.la 682> 683> #Load the syncprov overlay 684> moduleload syncprov.la 685> 686> # Accesslog database definitions 687> database hdb 688> suffix cn=accesslog 689> directory /db/accesslog 690> rootdn cn=accesslog 691> index default eq 692> index entryCSN,objectClass,reqEnd,reqResult,reqStart 693> 694> overlay syncprov 695> syncprov-nopresent TRUE 696> syncprov-reloadhint TRUE 697> 698> # Let the replica DN have limitless searches 699> limits dn.exact="cn=replicator,dc=symas,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited 700> 701> # Primary database definitions 702> database hdb 703> suffix "dc=symas,dc=com" 704> rootdn "cn=manager,dc=symas,dc=com" 705> 706> ## Whatever other configuration options are desired 707> 708> # syncprov specific indexing 709> index entryCSN eq 710> index entryUUID eq 711> 712> # syncrepl Provider for primary db 713> overlay syncprov 714> syncprov-checkpoint 1000 60 715> 716> # accesslog overlay definitions for primary db 717> overlay accesslog 718> logdb cn=accesslog 719> logops writes 720> logsuccess TRUE 721> # scan the accesslog DB every day, and purge entries older than 7 days 722> logpurge 07+00:00 01+00:00 723> 724> # Let the replica DN have limitless searches 725> limits dn.exact="cn=replicator,dc=symas,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited 726 727For more information, always consult the relevant man pages ({{slapo-accesslog}}(5) and {{slapd.conf}}(5)) 728 729 730H4: Delta-syncrepl Consumer configuration 731 732> # Replica database configuration 733> database hdb 734> suffix "dc=symas,dc=com" 735> rootdn "cn=manager,dc=symas,dc=com" 736> 737> ## Whatever other configuration bits for the replica, like indexing 738> ## that you want 739> 740> # syncrepl specific indices 741> index entryUUID eq 742> 743> # syncrepl directives 744> syncrepl rid=0 745> provider=ldap://ldapmaster.symas.com:389 746> bindmethod=simple 747> binddn="cn=replicator,dc=symas,dc=com" 748> credentials=secret 749> searchbase="dc=symas,dc=com" 750> logbase="cn=accesslog" 751> logfilter="(&(objectClass=auditWriteObject)(reqResult=0))" 752> schemachecking=on 753> type=refreshAndPersist 754> retry="60 +" 755> syncdata=accesslog 756> 757> # Refer updates to the master 758> updateref ldap://ldapmaster.symas.com 759 760 761The above configuration assumes that you have a replicator identity defined 762in your database that can be used to bind to the provider. In addition, 763all of the databases (primary, replica, and the accesslog 764storage database) should also have properly tuned {{DB_CONFIG}} files that meet 765your needs. 766 767Note: An accesslog database is unique to a given master. It should 768never be replicated. 769 770H3: N-Way Multi-Master 771 772For the following example we will be using 3 Master nodes. Keeping in line with 773{{B:test050-syncrepl-multimaster}} of the OpenLDAP test suite, we will be configuring 774{{slapd(8)}} via {{B:cn=config}} 775 776This sets up the config database: 777 778> dn: cn=config 779> objectClass: olcGlobal 780> cn: config 781> olcServerID: 1 782> 783> dn: olcDatabase={0}config,cn=config 784> objectClass: olcDatabaseConfig 785> olcDatabase: {0}config 786> olcRootPW: secret 787 788second and third servers will have a different olcServerID obviously: 789 790> dn: cn=config 791> objectClass: olcGlobal 792> cn: config 793> olcServerID: 2 794> 795> dn: olcDatabase={0}config,cn=config 796> objectClass: olcDatabaseConfig 797> olcDatabase: {0}config 798> olcRootPW: secret 799 800This sets up syncrepl as a provider (since these are all masters): 801 802> dn: cn=module,cn=config 803> objectClass: olcModuleList 804> cn: module 805> olcModulePath: /usr/local/libexec/openldap 806> olcModuleLoad: syncprov.la 807 808Now we setup the first Master Node (replace $URI1, $URI2 and $URI3 etc. with your actual ldap urls): 809 810> dn: cn=config 811> changetype: modify 812> replace: olcServerID 813> olcServerID: 1 $URI1 814> olcServerID: 2 $URI2 815> olcServerID: 3 $URI3 816> 817> dn: olcOverlay=syncprov,olcDatabase={0}config,cn=config 818> changetype: add 819> objectClass: olcOverlayConfig 820> objectClass: olcSyncProvConfig 821> olcOverlay: syncprov 822> 823> dn: olcDatabase={0}config,cn=config 824> changetype: modify 825> add: olcSyncRepl 826> olcSyncRepl: rid=001 provider=$URI1 binddn="cn=config" bindmethod=simple 827> credentials=secret searchbase="cn=config" type=refreshAndPersist 828> retry="5 5 300 5" timeout=1 829> olcSyncRepl: rid=002 provider=$URI2 binddn="cn=config" bindmethod=simple 830> credentials=secret searchbase="cn=config" type=refreshAndPersist 831> retry="5 5 300 5" timeout=1 832> olcSyncRepl: rid=003 provider=$URI3 binddn="cn=config" bindmethod=simple 833> credentials=secret searchbase="cn=config" type=refreshAndPersist 834> retry="5 5 300 5" timeout=1 835> - 836> add: olcMirrorMode 837> olcMirrorMode: TRUE 838 839Now start up the Master and a consumer/s, also add the above LDIF to the first consumer, second consumer etc. It will then replicate {{B:cn=config}}. You now have N-Way Multimaster on the config database. 840 841We still have to replicate the actual data, not just the config, so add to the master (all active and configured consumers/masters will pull down this config, as they are all syncing). Also, replace all {{${}}} variables with whatever is applicable to your setup: 842 843> dn: olcDatabase={1}$BACKEND,cn=config 844> objectClass: olcDatabaseConfig 845> objectClass: olc${BACKEND}Config 846> olcDatabase: {1}$BACKEND 847> olcSuffix: $BASEDN 848> olcDbDirectory: ./db 849> olcRootDN: $MANAGERDN 850> olcRootPW: $PASSWD 851> olcLimits: dn.exact="$MANAGERDN" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited 852> olcSyncRepl: rid=004 provider=$URI1 binddn="$MANAGERDN" bindmethod=simple 853> credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly 854> interval=00:00:00:10 retry="5 5 300 5" timeout=1 855> olcSyncRepl: rid=005 provider=$URI2 binddn="$MANAGERDN" bindmethod=simple 856> credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly 857> interval=00:00:00:10 retry="5 5 300 5" timeout=1 858> olcSyncRepl: rid=006 provider=$URI3 binddn="$MANAGERDN" bindmethod=simple 859> credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly 860> interval=00:00:00:10 retry="5 5 300 5" timeout=1 861> olcMirrorMode: TRUE 862> 863> dn: olcOverlay=syncprov,olcDatabase={1}${BACKEND},cn=config 864> changetype: add 865> objectClass: olcOverlayConfig 866> objectClass: olcSyncProvConfig 867> olcOverlay: syncprov 868 869Note: All of your servers' clocks must be tightly synchronized using 870e.g. NTP {{http://www.ntp.org/}}, atomic clock, or some other reliable 871time reference. 872 873Note: As stated in {{slapd-config}}(5), URLs specified in {{olcSyncRepl}} 874directives are the URLs of the servers from which to replicate. These 875must exactly match the URLs {{slapd}} listens on ({{-h}} in {{SECT:Command-Line Options}}). 876Otherwise slapd may attempt to replicate from itself, causing a loop. 877 878H3: MirrorMode 879 880MirrorMode configuration is actually very easy. If you have ever setup a normal 881slapd syncrepl provider, then the only change is the following two directives: 882 883> mirrormode on 884> serverID 1 885 886Note: You need to make sure that the {{serverID}} of each mirror node is 887different and add it as a global configuration option. 888 889H4: Mirror Node Configuration 890 891The first step is to configure the syncrepl provider the same as in the 892{{SECT:Set up the provider slapd}} section. 893 894Here's a specific cut down example using {{SECT:LDAP Sync Replication}} in 895{{refreshAndPersist}} mode: 896 897MirrorMode node 1: 898 899> # Global section 900> serverID 1 901> # database section 902> 903> # syncrepl directive 904> syncrepl rid=001 905> provider=ldap://ldap-sid2.example.com 906> bindmethod=simple 907> binddn="cn=mirrormode,dc=example,dc=com" 908> credentials=mirrormode 909> searchbase="dc=example,dc=com" 910> schemachecking=on 911> type=refreshAndPersist 912> retry="60 +" 913> 914> mirrormode on 915 916MirrorMode node 2: 917 918> # Global section 919> serverID 2 920> # database section 921> 922> # syncrepl directive 923> syncrepl rid=001 924> provider=ldap://ldap-sid1.example.com 925> bindmethod=simple 926> binddn="cn=mirrormode,dc=example,dc=com" 927> credentials=mirrormode 928> searchbase="dc=example,dc=com" 929> schemachecking=on 930> type=refreshAndPersist 931> retry="60 +" 932> 933> mirrormode on 934 935It's simple really; each MirrorMode node is setup {{B:exactly}} the same, except 936that the {{serverID}} is unique, and each consumer is pointed to 937the other server. 938 939H5: Failover Configuration 940 941There are generally 2 choices for this; 1. Hardware proxies/load-balancing or 942dedicated proxy software, 2. using a Back-LDAP proxy as a syncrepl provider 943 944A typical enterprise example might be: 945 946!import "dual_dc.png"; align="center"; title="MirrorMode Enterprise Configuration" 947FT[align="Center"] Figure X.Y: MirrorMode in a Dual Data Center Configuration 948 949H5: Normal Consumer Configuration 950 951This is exactly the same as the {{SECT:Set up the consumer slapd}} section. It 952can either setup in normal {{SECT:syncrepl replication}} mode, or in 953{{SECT:delta-syncrepl replication}} mode. 954 955H4: MirrorMode Summary 956 957You will now have a directory architecture that provides all of the 958consistency guarantees of single-master replication, while also providing the 959high availability of multi-master replication. 960 961 962H3: Syncrepl Proxy 963 964!import "push-based-complete.png"; align="center"; title="Syncrepl Proxy Mode" 965FT[align="Center"] Figure X.Y: Replacing slurpd 966 967The following example is for a self-contained push-based replication solution: 968 969> ####################################################################### 970> # Standard OpenLDAP Master/Provider 971> ####################################################################### 972> 973> include /usr/local/etc/openldap/schema/core.schema 974> include /usr/local/etc/openldap/schema/cosine.schema 975> include /usr/local/etc/openldap/schema/nis.schema 976> include /usr/local/etc/openldap/schema/inetorgperson.schema 977> 978> include /usr/local/etc/openldap/slapd.acl 979> 980> modulepath /usr/local/libexec/openldap 981> moduleload back_hdb.la 982> moduleload syncprov.la 983> moduleload back_monitor.la 984> moduleload back_ldap.la 985> 986> pidfile /usr/local/var/slapd.pid 987> argsfile /usr/local/var/slapd.args 988> 989> loglevel sync stats 990> 991> database hdb 992> suffix "dc=suretecsystems,dc=com" 993> directory /usr/local/var/openldap-data 994> 995> checkpoint 1024 5 996> cachesize 10000 997> idlcachesize 10000 998> 999> index objectClass eq 1000> # rest of indexes 1001> index default sub 1002> 1003> rootdn "cn=admin,dc=suretecsystems,dc=com" 1004> rootpw testing 1005> 1006> # syncprov specific indexing 1007> index entryCSN eq 1008> index entryUUID eq 1009> 1010> # syncrepl Provider for primary db 1011> overlay syncprov 1012> syncprov-checkpoint 1000 60 1013> 1014> # Let the replica DN have limitless searches 1015> limits dn.exact="cn=replicator,dc=suretecsystems,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited 1016> 1017> database monitor 1018> 1019> database config 1020> rootpw testing 1021> 1022> ############################################################################## 1023> # Consumer Proxy that pulls in data via Syncrepl and pushes out via slapd-ldap 1024> ############################################################################## 1025> 1026> database ldap 1027> # ignore conflicts with other databases, as we need to push out to same suffix 1028> hidden on 1029> suffix "dc=suretecsystems,dc=com" 1030> rootdn "cn=slapd-ldap" 1031> uri ldap://localhost:9012/ 1032> 1033> lastmod on 1034> 1035> # We don't need any access to this DSA 1036> restrict all 1037> 1038> acl-bind bindmethod=simple 1039> binddn="cn=replicator,dc=suretecsystems,dc=com" 1040> credentials=testing 1041> 1042> syncrepl rid=001 1043> provider=ldap://localhost:9011/ 1044> binddn="cn=replicator,dc=suretecsystems,dc=com" 1045> bindmethod=simple 1046> credentials=testing 1047> searchbase="dc=suretecsystems,dc=com" 1048> type=refreshAndPersist 1049> retry="5 5 300 5" 1050> 1051> overlay syncprov 1052 1053A replica configuration for this type of setup could be: 1054 1055> ####################################################################### 1056> # Standard OpenLDAP Slave without Syncrepl 1057> ####################################################################### 1058> 1059> include /usr/local/etc/openldap/schema/core.schema 1060> include /usr/local/etc/openldap/schema/cosine.schema 1061> include /usr/local/etc/openldap/schema/nis.schema 1062> include /usr/local/etc/openldap/schema/inetorgperson.schema 1063> 1064> include /usr/local/etc/openldap/slapd.acl 1065> 1066> modulepath /usr/local/libexec/openldap 1067> moduleload back_hdb.la 1068> moduleload syncprov.la 1069> moduleload back_monitor.la 1070> moduleload back_ldap.la 1071> 1072> pidfile /usr/local/var/slapd.pid 1073> argsfile /usr/local/var/slapd.args 1074> 1075> loglevel sync stats 1076> 1077> database hdb 1078> suffix "dc=suretecsystems,dc=com" 1079> directory /usr/local/var/openldap-slave/data 1080> 1081> checkpoint 1024 5 1082> cachesize 10000 1083> idlcachesize 10000 1084> 1085> index objectClass eq 1086> # rest of indexes 1087> index default sub 1088> 1089> rootdn "cn=admin,dc=suretecsystems,dc=com" 1090> rootpw testing 1091> 1092> # Let the replica DN have limitless searches 1093> limits dn.exact="cn=replicator,dc=suretecsystems,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited 1094> 1095> updatedn "cn=replicator,dc=suretecsystems,dc=com" 1096> 1097> # Refer updates to the master 1098> updateref ldap://localhost:9011 1099> 1100> database monitor 1101> 1102> database config 1103> rootpw testing 1104 1105You can see we use the {{updatedn}} directive here and example ACLs ({{F:usr/local/etc/openldap/slapd.acl}}) for this could be: 1106 1107> # Give the replica DN unlimited read access. This ACL may need to be 1108> # merged with other ACL statements. 1109> 1110> access to * 1111> by dn.base="cn=replicator,dc=suretecsystems,dc=com" write 1112> by * break 1113> 1114> access to dn.base="" 1115> by * read 1116> 1117> access to dn.base="cn=Subschema" 1118> by * read 1119> 1120> access to dn.subtree="cn=Monitor" 1121> by dn.exact="uid=admin,dc=suretecsystems,dc=com" write 1122> by users read 1123> by * none 1124> 1125> access to * 1126> by self write 1127> by * read 1128 1129In order to support more replicas, just add more {{database ldap}} sections and 1130increment the {{syncrepl rid}} number accordingly. 1131 1132Note: You must populate the Master and Slave directories with the same data, 1133unlike when using normal Syncrepl 1134 1135If you do not have access to modify the master directory configuration you can 1136configure a standalone ldap proxy, which might look like: 1137 1138!import "push-based-standalone.png"; align="center"; title="Syncrepl Standalone Proxy Mode" 1139FT[align="Center"] Figure X.Y: Replacing slurpd with a standalone version 1140 1141The following configuration is an example of a standalone LDAP Proxy: 1142 1143> include /usr/local/etc/openldap/schema/core.schema 1144> include /usr/local/etc/openldap/schema/cosine.schema 1145> include /usr/local/etc/openldap/schema/nis.schema 1146> include /usr/local/etc/openldap/schema/inetorgperson.schema 1147> 1148> include /usr/local/etc/openldap/slapd.acl 1149> 1150> modulepath /usr/local/libexec/openldap 1151> moduleload syncprov.la 1152> moduleload back_ldap.la 1153> 1154> ############################################################################## 1155> # Consumer Proxy that pulls in data via Syncrepl and pushes out via slapd-ldap 1156> ############################################################################## 1157> 1158> database ldap 1159> # ignore conflicts with other databases, as we need to push out to same suffix 1160> hidden on 1161> suffix "dc=suretecsystems,dc=com" 1162> rootdn "cn=slapd-ldap" 1163> uri ldap://localhost:9012/ 1164> 1165> lastmod on 1166> 1167> # We don't need any access to this DSA 1168> restrict all 1169> 1170> acl-bind bindmethod=simple 1171> binddn="cn=replicator,dc=suretecsystems,dc=com" 1172> credentials=testing 1173> 1174> syncrepl rid=001 1175> provider=ldap://localhost:9011/ 1176> binddn="cn=replicator,dc=suretecsystems,dc=com" 1177> bindmethod=simple 1178> credentials=testing 1179> searchbase="dc=suretecsystems,dc=com" 1180> type=refreshAndPersist 1181> retry="5 5 300 5" 1182> 1183> overlay syncprov 1184 1185As you can see, you can let your imagination go wild using Syncrepl and 1186{{slapd-ldap(8)}} tailoring your replication to fit your specific network 1187topology. 1188
