1# $OpenLDAP$ 2# Copyright 1999-2014 The OpenLDAP Foundation, All Rights Reserved. 3# COPYING RESTRICTIONS APPLY, see COPYRIGHT. 4 5H1: Replication 6 7Replicated directories are a fundamental requirement for delivering a 8resilient enterprise deployment. 9 10{{PRD:OpenLDAP}} has various configuration options for creating a replicated 11directory. In previous releases, replication was discussed in terms of 12a {{master}} server and some number of {{slave}} servers. A master 13accepted directory updates from other clients, and a slave only 14accepted updates from a (single) master. The replication structure 15was rigidly defined and any particular database could only fulfill 16a single role, either master or slave. 17 18As OpenLDAP now supports a wide variety of replication topologies, these 19terms have been deprecated in favor of {{provider}} and 20{{consumer}}: A provider replicates directory updates to consumers; 21consumers receive replication updates from providers. Unlike the 22rigidly defined master/slave relationships, provider/consumer roles 23are quite fluid: replication updates received in a consumer can be 24further propagated by that consumer to other servers, so a consumer 25can also act simultaneously as a provider. Also, a consumer need not 26be an actual LDAP server; it may be just an LDAP client. 27 28The following sections will describe the replication technology and 29discuss the various replication options that are available. 30 31H2: Replication Technology 32 33H3: LDAP Sync Replication 34 35The {{TERM:LDAP Sync}} Replication engine, {{TERM:syncrepl}} for 36short, is a consumer-side replication engine that enables the 37consumer {{TERM:LDAP}} server to maintain a shadow copy of a 38{{TERM:DIT}} fragment. A syncrepl engine resides at the consumer 39and executes as one of the {{slapd}}(8) threads. It creates and maintains a 40consumer replica by connecting to the replication provider to perform 41the initial DIT content load followed either by periodic content 42polling or by timely updates upon content changes. 43 44Syncrepl uses the LDAP Content Synchronization protocol (or LDAP Sync for 45short) as the replica synchronization protocol. LDAP Sync provides 46a stateful replication which supports both pull-based and push-based 47synchronization and does not mandate the use of a history store. 48In pull-based replication the consumer periodically 49polls the provider for updates. In push-based replication the consumer 50listens for updates that are sent by the provider in realtime. Since the 51protocol does not require a history store, the provider does not need to 52maintain any log of updates it has received (Note 53that the syncrepl engine is extensible and additional replication 54protocols may be supported in the future.). 55 56Syncrepl keeps track of the status of the replication content by 57maintaining and exchanging synchronization cookies. Because the 58syncrepl consumer and provider maintain their content status, the 59consumer can poll the provider content to perform incremental 60synchronization by asking for the entries required to make the 61consumer replica up-to-date with the provider content. Syncrepl 62also enables convenient management of replicas by maintaining replica 63status. The consumer replica can be constructed from a consumer-side 64or a provider-side backup at any synchronization status. Syncrepl 65can automatically resynchronize the consumer replica up-to-date 66with the current provider content. 67 68Syncrepl supports both pull-based and push-based synchronization. 69In its basic refreshOnly synchronization mode, the provider uses 70pull-based synchronization where the consumer servers need not be 71tracked and no history information is maintained. The information 72required for the provider to process periodic polling requests is 73contained in the synchronization cookie of the request itself. To 74optimize the pull-based synchronization, syncrepl utilizes the 75present phase of the LDAP Sync protocol as well as its delete phase, 76instead of falling back on frequent full reloads. To further optimize 77the pull-based synchronization, the provider can maintain a per-scope 78session log as a history store. In its refreshAndPersist mode of 79synchronization, the provider uses a push-based synchronization. 80The provider keeps track of the consumer servers that have requested 81a persistent search and sends them necessary updates as the provider 82replication content gets modified. 83 84With syncrepl, a consumer server can create a replica without 85changing the provider's configurations and without restarting the 86provider server, if the consumer server has appropriate access 87privileges for the DIT fragment to be replicated. The consumer 88server can stop the replication also without the need for provider-side 89changes and restart. 90 91Syncrepl supports partial, sparse, and fractional replications. The shadow 92DIT fragment is defined by a general search criteria consisting of 93base, scope, filter, and attribute list. The replica content is 94also subject to the access privileges of the bind identity of the 95syncrepl replication connection. 96 97 98H4: The LDAP Content Synchronization Protocol 99 100The LDAP Sync protocol allows a client to maintain a synchronized 101copy of a DIT fragment. The LDAP Sync operation is defined as a set 102of controls and other protocol elements which extend the LDAP search 103operation. This section introduces the LDAP Content Sync protocol 104only briefly. For more information, refer to {{REF:RFC4533}}. 105 106The LDAP Sync protocol supports both polling and listening for changes 107by defining two respective synchronization operations: 108{{refreshOnly}} and {{refreshAndPersist}}. Polling is implemented 109by the {{refreshOnly}} operation. The consumer 110polls the provider using an LDAP Search request with an LDAP Sync 111control attached. The consumer copy is synchronized 112to the provider copy at the time of polling using the information 113returned in the search. The provider finishes the 114search operation by returning {{SearchResultDone}} at the end of 115the search operation as in the normal search. Listening is 116implemented by the {{refreshAndPersist}} operation. As the name 117implies, it begins with a search, like refreshOnly. Instead of 118finishing the search after returning all entries currently matching 119the search criteria, the synchronization search remains persistent 120in the provider. Subsequent updates to the synchronization content 121in the provider cause additional entry updates to be sent to the 122consumer. 123 124The {{refreshOnly}} operation and the refresh stage of the 125{{refreshAndPersist}} operation can be performed with a present 126phase or a delete phase. 127 128In the present phase, the provider sends the consumer the entries updated 129within the search scope since the last synchronization. The provider 130sends all requested attributes, be they changed or not, of the updated 131entries. For each unchanged entry which remains in the scope, the 132provider sends a present message consisting only of the name of the 133entry and the synchronization control representing state present. 134The present message does not contain any attributes of the entry. 135After the consumer receives all update and present entries, it can 136reliably determine the new consumer copy by adding the entries added 137to the provider, by replacing the entries modified at the provider, and 138by deleting entries in the consumer copy which have not been updated 139nor specified as being present at the provider. 140 141The transmission of the updated entries in the delete phase is the 142same as in the present phase. The provider sends all the requested 143attributes of the entries updated within the search scope since the 144last synchronization to the consumer. In the delete phase, however, 145the provider sends a delete message for each entry deleted from the 146search scope, instead of sending present messages. The delete 147message consists only of the name of the entry and the synchronization 148control representing state delete. The new consumer copy can be 149determined by adding, modifying, and removing entries according to 150the synchronization control attached to the {{SearchResultEntry}} 151message. 152 153In the case that the LDAP Sync provider maintains a history store and 154can determine which entries are scoped out of the consumer copy since 155the last synchronization time, the provider can use the delete phase. 156If the provider does not maintain any history store, cannot determine 157the scoped-out entries from the history store, or the history store 158does not cover the outdated synchronization state of the consumer, 159the provider should use the present phase. The use of the present 160phase is much more efficient than a full content reload in terms 161of the synchronization traffic. To reduce the synchronization 162traffic further, the LDAP Sync protocol also provides several 163optimizations such as the transmission of the normalized {{EX:entryUUID}}s 164and the transmission of multiple {{EX:entryUUIDs}} in a single 165{{syncIdSet}} message. 166 167At the end of the {{refreshOnly}} synchronization, the provider sends 168a synchronization cookie to the consumer as a state indicator of the 169consumer copy after the synchronization is completed. The consumer 170will present the received cookie when it requests the next incremental 171synchronization to the provider. 172 173When {{refreshAndPersist}} synchronization is used, the provider sends 174a synchronization cookie at the end of the refresh stage by sending 175a Sync Info message with refreshDone=TRUE. It also sends a 176synchronization cookie by attaching it to {{SearchResultEntry}} 177messages generated in the persist stage of the synchronization search. During 178the persist stage, the provider can also send a Sync Info message 179containing the synchronization cookie at any time the provider wants 180to update the consumer-side state indicator. 181 182In the LDAP Sync protocol, entries are uniquely identified by the 183{{EX:entryUUID}} attribute value. It can function as a reliable 184identifier of the entry. The DN of the entry, on the other hand, 185can be changed over time and hence cannot be considered as the 186reliable identifier. The {{EX:entryUUID}} is attached to each 187{{SearchResultEntry}} or {{SearchResultReference}} as a part of the 188synchronization control. 189 190H4: Syncrepl Details 191 192The syncrepl engine utilizes both the {{refreshOnly}} and the 193{{refreshAndPersist}} operations of the LDAP Sync protocol. If a 194syncrepl specification is included in a database definition, 195{{slapd}}(8) launches a syncrepl engine as a {{slapd}}(8) thread 196and schedules its execution. If the {{refreshOnly}} operation is 197specified, the syncrepl engine will be rescheduled at the interval 198time after a synchronization operation is completed. If the 199{{refreshAndPersist}} operation is specified, the engine will remain 200active and process the persistent synchronization messages from the 201provider. 202 203The syncrepl engine utilizes both the present phase and the delete 204phase of the refresh synchronization. It is possible to configure 205a session log in the provider which stores the 206{{EX:entryUUID}}s of a finite number of entries deleted from a 207database. Multiple replicas share the same session log. The syncrepl 208engine uses the 209delete phase if the session log is present and the state of the 210consumer server is recent enough that no session log entries are 211truncated after the last synchronization of the client. The syncrepl 212engine uses the present phase if no session log is configured for 213the replication content or if the consumer replica is too outdated 214to be covered by the session log. The current design of the session 215log store is memory based, so the information contained in the 216session log is not persistent over multiple provider invocations. 217It is not currently supported to access the session log store by 218using LDAP operations. It is also not currently supported to impose 219access control to the session log. 220 221As a further optimization, even in the case the synchronization 222search is not associated with any session log, no entries will be 223transmitted to the consumer server when there has been no update 224in the replication context. 225 226The syncrepl engine, which is a consumer-side replication engine, 227can work with any backends. The LDAP Sync provider can be configured 228as an overlay on any backend, but works best with the {{back-bdb}} 229or {{back-hdb}} backend. 230 231The LDAP Sync provider maintains a {{EX:contextCSN}} for each 232database as the current synchronization state indicator of the 233provider content. It is the largest {{EX:entryCSN}} in the provider 234context such that no transactions for an entry having smaller 235{{EX:entryCSN}} value remains outstanding. The {{EX:contextCSN}} 236could not just be set to the largest issued {{EX:entryCSN}} because 237{{EX:entryCSN}} is obtained before a transaction starts and 238transactions are not committed in the issue order. 239 240The provider stores the {{EX:contextCSN}} of a context in the 241{{EX:contextCSN}} attribute of the context suffix entry. The attribute 242is not written to the database after every update operation though; 243instead it is maintained primarily in memory. At database start 244time the provider reads the last saved {{EX:contextCSN}} into memory 245and uses the in-memory copy exclusively thereafter. By default, 246changes to the {{EX:contextCSN}} as a result of database updates 247will not be written to the database until the server is cleanly 248shut down. A checkpoint facility exists to cause the {{EX:contextCSN}} to 249be written out more frequently if desired. 250 251Note that at startup time, if the provider is unable to read a 252{{EX:contextCSN}} from the suffix entry, it will scan the entire 253database to determine the value, and this scan may take quite a 254long time on a large database. When a {{EX:contextCSN}} value is 255read, the database will still be scanned for any {{EX:entryCSN}} 256values greater than it, to make sure the {{EX:contextCSN}} value 257truly reflects the greatest committed {{EX:entryCSN}} in the database. 258On databases which support inequality indexing, setting an eq index 259on the {{EX:entryCSN}} attribute and configuring {{contextCSN}} 260checkpoints will greatly speed up this scanning step. 261 262If no {{EX:contextCSN}} can be determined by reading and scanning 263the database, a new value will be generated. Also, if scanning the 264database yielded a greater {{EX:entryCSN}} than was previously 265recorded in the suffix entry's {{EX:contextCSN}} attribute, a 266checkpoint will be immediately written with the new value. 267 268The consumer also stores its replica state, which is the provider's 269{{EX:contextCSN}} received as a synchronization cookie, in the 270{{EX:contextCSN}} attribute of the suffix entry. The replica state 271maintained by a consumer server is used as the synchronization state 272indicator when it performs subsequent incremental synchronization 273with the provider server. It is also used as a provider-side 274synchronization state indicator when it functions as a secondary 275provider server in a cascading replication configuration. Since 276the consumer and provider state information are maintained in the 277same location within their respective databases, any consumer can 278be promoted to a provider (and vice versa) without any special 279actions. 280 281Because a general search filter can be used in the syncrepl 282specification, some entries in the context may be omitted from the 283synchronization content. The syncrepl engine creates a glue entry 284to fill in the holes in the replica context if any part of the 285replica content is subordinate to the holes. The glue entries will 286not be returned in the search result unless {{ManageDsaIT}} control 287is provided. 288 289Also as a consequence of the search filter used in the syncrepl 290specification, it is possible for a modification to remove an entry 291from the replication scope even though the entry has not been deleted 292on the provider. Logically the entry must be deleted on the consumer 293but in {{refreshOnly}} mode the provider cannot detect and propagate 294this change without the use of the session log on the provider. 295 296For configuration, please see the {{SECT:Syncrepl}} section. 297 298 299H2: Deployment Alternatives 300 301While the LDAP Sync specification only defines a narrow scope for replication, 302the OpenLDAP implementation is extremely flexible and supports a variety of 303operating modes to handle other scenarios not explicitly addressed in the spec. 304 305 306H3: Delta-syncrepl replication 307 308* Disadvantages of LDAP Sync replication: 309 310LDAP Sync replication is an object-based replication mechanism. 311When any attribute value in a replicated object is changed on the provider, 312each consumer fetches and processes the complete changed object, including 313{{B:both the changed and unchanged attribute values}} during replication. 314One advantage of this approach is that when multiple changes occur to 315a single object, the precise sequence of those changes need not be preserved; 316only the final state of the entry is significant. But this approach 317may have drawbacks when the usage pattern involves single changes to 318multiple objects. 319 320For example, suppose you have a database consisting of 102,400 objects of 1 KB 321each. Further, suppose you routinely run a batch job to change the value of 322a single two-byte attribute value that appears in each of the 102,400 objects 323on the master. Not counting LDAP and TCP/IP protocol overhead, each time you 324run this job each consumer will transfer and process {{B:100 MB}} of data to 325process {{B:200KB of changes!}} 326 32799.98% of the data that is transmitted and processed in a case like this will 328be redundant, since it represents values that did not change. This is a waste 329of valuable transmission and processing bandwidth and can cause an unacceptable 330replication backlog to develop. While this situation is extreme, it serves to 331demonstrate a very real problem that is encountered in some LDAP deployments. 332 333 334* Where Delta-syncrepl comes in: 335 336Delta-syncrepl, a changelog-based variant of syncrepl, is designed to address 337situations like the one described above. Delta-syncrepl works by maintaining a 338changelog of a selectable depth in a separate database on the provider. The replication consumer 339checks the changelog for the changes it needs and, as long as 340the changelog contains the needed changes, the consumer fetches the changes 341from the changelog and applies them to its database. If, however, a replica 342is too far out of sync (or completely empty), conventional syncrepl is used to 343bring it up to date and replication then switches back to the delta-syncrepl 344mode. 345 346Note: since the database state is stored in both the changelog DB and the 347main DB on the provider, it is important to backup/restore both the changelog 348DB and the main DB using slapcat/slapadd when restoring a DB or copying 349it to another machine. 350 351For configuration, please see the {{SECT:Delta-syncrepl}} section. 352 353 354H3: N-Way Multi-Master replication 355 356Multi-Master replication is a replication technique using Syncrepl to replicate 357data to multiple provider ("Master") Directory servers. 358 359H4: Valid Arguments for Multi-Master replication 360 361* If any provider fails, other providers will continue to accept updates 362* Avoids a single point of failure 363* Providers can be located in several physical sites i.e. distributed across 364the network/globe. 365* Good for Automatic failover/High Availability 366 367H4: Invalid Arguments for Multi-Master replication 368 369(These are often claimed to be advantages of Multi-Master replication but 370those claims are false): 371 372* It has {{B:NOTHING}} to do with load balancing 373* Providers {{B:must}} propagate writes to {{B:all}} the other servers, which 374means the network traffic and write load spreads across all 375of the servers the same as for single-master. 376* Server utilization and performance are at best identical for 377Multi-Master and Single-Master replication; at worst Single-Master is 378superior because indexing can be tuned differently to optimize for the 379different usage patterns between the provider and the consumers. 380 381H4: Arguments against Multi-Master replication 382 383* Breaks the data consistency guarantees of the directory model 384* {{URL:http://www.openldap.org/faq/data/cache/1240.html}} 385* If connectivity with a provider is lost because of a network partition, then 386"automatic failover" can just compound the problem 387* Typically, a particular machine cannot distinguish between losing contact 388 with a peer because that peer crashed, or because the network link has failed 389* If a network is partitioned and multiple clients start writing to each of the 390"masters" then reconciliation will be a pain; it may be best to simply deny 391writes to the clients that are partitioned from the single provider 392 393 394For configuration, please see the {{SECT:N-Way Multi-Master}} section below 395 396H3: MirrorMode replication 397 398MirrorMode is a hybrid configuration that provides all of the consistency 399guarantees of single-master replication, while also providing the high 400availability of multi-master. In MirrorMode two providers are set up to 401replicate from each other (as a multi-master configuration), but an 402external frontend is employed to direct all writes to only one of 403the two servers. The second provider will only be used for writes if 404the first provider crashes, at which point the frontend will switch to 405directing all writes to the second provider. When a crashed provider is 406repaired and restarted it will automatically catch up to any changes 407on the running provider and resync. 408 409H4: Arguments for MirrorMode 410 411* Provides a high-availability (HA) solution for directory writes (replicas handle reads) 412* As long as one provider is operational, writes can safely be accepted 413* Provider nodes replicate from each other, so they are always up to date and 414can be ready to take over (hot standby) 415* Syncrepl also allows the provider nodes to re-synchronize after any downtime 416 417 418H4: Arguments against MirrorMode 419 420* MirrorMode is not what is termed as a Multi-Master solution. This is because 421writes have to go to just one of the mirror nodes at a time 422* MirrorMode can be termed as Active-Active Hot-Standby, therefore an external 423server (slapd in proxy mode) or device (hardware load balancer) 424is needed to manage which provider is currently active 425* Backups are managed slightly differently 426- If backing up the Berkeley database itself and periodically backing up the 427transaction log files, then the same member of the mirror pair needs to be 428used to collect logfiles until the next database backup is taken 429 430For configuration, please see the {{SECT:MirrorMode}} section below 431 432 433H3: Syncrepl Proxy Mode 434 435While the LDAP Sync protocol supports both pull- and push-based replication, 436the push mode (refreshAndPersist) must still be initiated from the consumer 437before the provider can begin pushing changes. In some network configurations, 438particularly where firewalls restrict the direction in which connections 439can be made, a provider-initiated push mode may be needed. 440 441This mode can be configured with the aid of the LDAP Backend 442({{SECT: Backends}} and {{slapd-ldap(8)}}). Instead of running the 443syncrepl engine on the actual consumer, a slapd-ldap proxy is set up 444near (or collocated with) the provider that points to the consumer, 445and the syncrepl engine runs on the proxy. 446 447For configuration, please see the {{SECT:Syncrepl Proxy}} section. 448 449H4: Replacing Slurpd 450 451The old {{slurpd}} mechanism only operated in provider-initiated 452push mode. Slurpd replication was deprecated in favor of Syncrepl 453replication and has been completely removed from OpenLDAP 2.4. 454 455The slurpd daemon was the original replication mechanism inherited from 456UMich's LDAP and operated in push mode: the master pushed changes to the 457slaves. It was replaced for many reasons, in brief: 458 459 * It was not reliable 460 ** It was extremely sensitive to the ordering of records in the replog 461 ** It could easily go out of sync, at which point manual intervention was 462 required to resync the slave database with the master directory 463 ** It wasn't very tolerant of unavailable servers. If a slave went down 464 for a long time, the replog could grow to a size that was too large for 465 slurpd to process 466 * It only worked in push mode 467 * It required stopping and restarting the master to add new slaves 468 * It only supported single master replication 469 470Syncrepl has none of those weaknesses: 471 472 * Syncrepl is self-synchronizing; you can start with a consumer database 473 in any state from totally empty to fully synced and it will automatically 474 do the right thing to achieve and maintain synchronization 475 ** It is completely insensitive to the order in which changes occur 476 ** It guarantees convergence between the consumer and the provider 477 content without manual intervention 478 ** It can resynchronize regardless of how long a consumer stays out 479 of contact with the provider 480 * Syncrepl can operate in either direction 481 * Consumers can be added at any time without touching anything on the 482 provider 483 * Multi-master replication is supported 484 485 486H2: Configuring the different replication types 487 488H3: Syncrepl 489 490H4: Syncrepl configuration 491 492Because syncrepl is a consumer-side replication engine, the syncrepl 493specification is defined in {{slapd.conf}}(5) of the consumer 494server, not in the provider server's configuration file. The initial 495loading of the replica content can be performed either by starting 496the syncrepl engine with no synchronization cookie or by populating 497the consumer replica by loading an {{TERM:LDIF}} file dumped as a 498backup at the provider. 499 500When loading from a backup, it is not required to perform the initial 501loading from the up-to-date backup of the provider content. The 502syncrepl engine will automatically synchronize the initial consumer 503replica to the current provider content. As a result, it is not 504required to stop the provider server in order to avoid the replica 505inconsistency caused by the updates to the provider content during 506the content backup and loading process. 507 508When replicating a large scale directory, especially in a bandwidth 509constrained environment, it is advised to load the consumer replica 510from a backup instead of performing a full initial load using 511syncrepl. 512 513 514H4: Set up the provider slapd 515 516The provider is implemented as an overlay, so the overlay itself 517must first be configured in {{slapd.conf}}(5) before it can be 518used. The provider has only two configuration directives, for setting 519checkpoints on the {{EX:contextCSN}} and for configuring the session 520log. Because the LDAP Sync search is subject to access control, 521proper access control privileges should be set up for the replicated 522content. 523 524The {{EX:contextCSN}} checkpoint is configured by the 525 526> syncprov-checkpoint <ops> <minutes> 527 528directive. Checkpoints are only tested after successful write 529operations. If {{<ops>}} operations or more than {{<minutes>}} 530time has passed since the last checkpoint, a new checkpoint is 531performed. 532 533The session log is configured by the 534 535> syncprov-sessionlog <size> 536 537directive, where {{<size>}} is the maximum number of session log 538entries the session log can record. When a session log is configured, 539it is automatically used for all LDAP Sync searches within the 540database. 541 542Note that using the session log requires searching on the {{entryUUID}} 543attribute. Setting an eq index on this attribute will greatly benefit 544the performance of the session log on the provider. 545 546A more complete example of the {{slapd.conf}}(5) content is thus: 547 548> database bdb 549> suffix dc=Example,dc=com 550> rootdn dc=Example,dc=com 551> directory /var/ldap/db 552> index objectclass,entryCSN,entryUUID eq 553> 554> overlay syncprov 555> syncprov-checkpoint 100 10 556> syncprov-sessionlog 100 557 558 559H4: Set up the consumer slapd 560 561The syncrepl replication is specified in the database section of 562{{slapd.conf}}(5) for the replica context. The syncrepl engine 563is backend independent and the directive can be defined with any 564database type. 565 566> database hdb 567> suffix dc=Example,dc=com 568> rootdn dc=Example,dc=com 569> directory /var/ldap/db 570> index objectclass,entryCSN,entryUUID eq 571> 572> syncrepl rid=123 573> provider=ldap://provider.example.com:389 574> type=refreshOnly 575> interval=01:00:00:00 576> searchbase="dc=example,dc=com" 577> filter="(objectClass=organizationalPerson)" 578> scope=sub 579> attrs="cn,sn,ou,telephoneNumber,title,l" 580> schemachecking=off 581> bindmethod=simple 582> binddn="cn=syncuser,dc=example,dc=com" 583> credentials=secret 584 585In this example, the consumer will connect to the provider {{slapd}}(8) 586at port 389 of {{FILE:ldap://provider.example.com}} to perform a 587polling ({{refreshOnly}}) mode of synchronization once a day. It 588will bind as {{EX:cn=syncuser,dc=example,dc=com}} using simple 589authentication with password "secret". Note that the access control 590privilege of {{EX:cn=syncuser,dc=example,dc=com}} should be set 591appropriately in the provider to retrieve the desired replication 592content. Also the search limits must be high enough on the provider 593to allow the syncuser to retrieve a complete copy of the requested 594content. The consumer uses the rootdn to write to its database so 595it always has full permissions to write all content. 596 597The synchronization search in the above example will search for the 598entries whose objectClass is organizationalPerson in the entire 599subtree rooted at {{EX:dc=example,dc=com}}. The requested attributes 600are {{EX:cn}}, {{EX:sn}}, {{EX:ou}}, {{EX:telephoneNumber}}, 601{{EX:title}}, and {{EX:l}}. The schema checking is turned off, so 602that the consumer {{slapd}}(8) will not enforce entry schema 603checking when it processes updates from the provider {{slapd}}(8). 604 605For more detailed information on the syncrepl directive, see the 606{{SECT:syncrepl}} section of {{SECT:The slapd Configuration File}} 607chapter of this admin guide. 608 609 610H4: Start the provider and the consumer slapd 611 612The provider {{slapd}}(8) is not required to be restarted. 613{{contextCSN}} is automatically generated as needed: it might be 614originally contained in the {{TERM:LDIF}} file, generated by 615{{slapadd}} (8), generated upon changes in the context, or generated 616when the first LDAP Sync search arrives at the provider. If an 617LDIF file is being loaded which did not previously contain the 618{{contextCSN}}, the {{-w}} option should be used with {{slapadd}} 619(8) to cause it to be generated. This will allow the server to 620startup a little quicker the first time it runs. 621 622When starting a consumer {{slapd}}(8), it is possible to provide 623a synchronization cookie as the {{-c cookie}} command line option 624in order to start the synchronization from a specific state. The 625cookie is a comma separated list of name=value pairs. Currently 626supported syncrepl cookie fields are {{csn=<csn>}} and {{rid=<rid>}}. 627{{<csn>}} represents the current synchronization state of the 628consumer replica. {{<rid>}} identifies a consumer replica locally 629within the consumer server. It is used to relate the cookie to the 630syncrepl definition in {{slapd.conf}}(5) which has the matching 631replica identifier. The {{<rid>}} must have no more than 3 decimal 632digits. The command line cookie overrides the synchronization 633cookie stored in the consumer replica database. 634 635 636H3: Delta-syncrepl 637 638H4: Delta-syncrepl Provider configuration 639 640Setting up delta-syncrepl requires configuration changes on both the master and 641replica servers: 642 643> # Give the replica DN unlimited read access. This ACL needs to be 644> # merged with other ACL statements, and/or moved within the scope 645> # of a database. The "by * break" portion causes evaluation of 646> # subsequent rules. See slapd.access(5) for details. 647> access to * 648> by dn.base="cn=replicator,dc=symas,dc=com" read 649> by * break 650> 651> # Set the module path location 652> modulepath /opt/symas/lib/openldap 653> 654> # Load the hdb backend 655> moduleload back_hdb.la 656> 657> # Load the accesslog overlay 658> moduleload accesslog.la 659> 660> #Load the syncprov overlay 661> moduleload syncprov.la 662> 663> # Accesslog database definitions 664> database hdb 665> suffix cn=accesslog 666> directory /db/accesslog 667> rootdn cn=accesslog 668> index default eq 669> index entryCSN,objectClass,reqEnd,reqResult,reqStart 670> 671> overlay syncprov 672> syncprov-nopresent TRUE 673> syncprov-reloadhint TRUE 674> 675> # Let the replica DN have limitless searches 676> limits dn.exact="cn=replicator,dc=symas,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited 677> 678> # Primary database definitions 679> database hdb 680> suffix "dc=symas,dc=com" 681> rootdn "cn=manager,dc=symas,dc=com" 682> 683> ## Whatever other configuration options are desired 684> 685> # syncprov specific indexing 686> index entryCSN eq 687> index entryUUID eq 688> 689> # syncrepl Provider for primary db 690> overlay syncprov 691> syncprov-checkpoint 1000 60 692> 693> # accesslog overlay definitions for primary db 694> overlay accesslog 695> logdb cn=accesslog 696> logops writes 697> logsuccess TRUE 698> # scan the accesslog DB every day, and purge entries older than 7 days 699> logpurge 07+00:00 01+00:00 700> 701> # Let the replica DN have limitless searches 702> limits dn.exact="cn=replicator,dc=symas,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited 703 704For more information, always consult the relevant man pages ({{slapo-accesslog}}(5) and {{slapd.conf}}(5)) 705 706 707H4: Delta-syncrepl Consumer configuration 708 709> # Replica database configuration 710> database hdb 711> suffix "dc=symas,dc=com" 712> rootdn "cn=manager,dc=symas,dc=com" 713> 714> ## Whatever other configuration bits for the replica, like indexing 715> ## that you want 716> 717> # syncrepl specific indices 718> index entryUUID eq 719> 720> # syncrepl directives 721> syncrepl rid=0 722> provider=ldap://ldapmaster.symas.com:389 723> bindmethod=simple 724> binddn="cn=replicator,dc=symas,dc=com" 725> credentials=secret 726> searchbase="dc=symas,dc=com" 727> logbase="cn=accesslog" 728> logfilter="(&(objectClass=auditWriteObject)(reqResult=0))" 729> schemachecking=on 730> type=refreshAndPersist 731> retry="60 +" 732> syncdata=accesslog 733> 734> # Refer updates to the master 735> updateref ldap://ldapmaster.symas.com 736 737 738The above configuration assumes that you have a replicator identity defined 739in your database that can be used to bind to the provider. In addition, 740all of the databases (primary, replica, and the accesslog 741storage database) should also have properly tuned {{DB_CONFIG}} files that meet 742your needs. 743 744 745H3: N-Way Multi-Master 746 747For the following example we will be using 3 Master nodes. Keeping in line with 748{{B:test050-syncrepl-multimaster}} of the OpenLDAP test suite, we will be configuring 749{{slapd(8)}} via {{B:cn=config}} 750 751This sets up the config database: 752 753> dn: cn=config 754> objectClass: olcGlobal 755> cn: config 756> olcServerID: 1 757> 758> dn: olcDatabase={0}config,cn=config 759> objectClass: olcDatabaseConfig 760> olcDatabase: {0}config 761> olcRootPW: secret 762 763second and third servers will have a different olcServerID obviously: 764 765> dn: cn=config 766> objectClass: olcGlobal 767> cn: config 768> olcServerID: 2 769> 770> dn: olcDatabase={0}config,cn=config 771> objectClass: olcDatabaseConfig 772> olcDatabase: {0}config 773> olcRootPW: secret 774 775This sets up syncrepl as a provider (since these are all masters): 776 777> dn: cn=module,cn=config 778> objectClass: olcModuleList 779> cn: module 780> olcModulePath: /usr/local/libexec/openldap 781> olcModuleLoad: syncprov.la 782 783Now we setup the first Master Node (replace $URI1, $URI2 and $URI3 etc. with your actual ldap urls): 784 785> dn: cn=config 786> changetype: modify 787> replace: olcServerID 788> olcServerID: 1 $URI1 789> olcServerID: 2 $URI2 790> olcServerID: 3 $URI3 791> 792> dn: olcOverlay=syncprov,olcDatabase={0}config,cn=config 793> changetype: add 794> objectClass: olcOverlayConfig 795> objectClass: olcSyncProvConfig 796> olcOverlay: syncprov 797> 798> dn: olcDatabase={0}config,cn=config 799> changetype: modify 800> add: olcSyncRepl 801> olcSyncRepl: rid=001 provider=$URI1 binddn="cn=config" bindmethod=simple 802> credentials=secret searchbase="cn=config" type=refreshAndPersist 803> retry="5 5 300 5" timeout=1 804> olcSyncRepl: rid=002 provider=$URI2 binddn="cn=config" bindmethod=simple 805> credentials=secret searchbase="cn=config" type=refreshAndPersist 806> retry="5 5 300 5" timeout=1 807> olcSyncRepl: rid=003 provider=$URI3 binddn="cn=config" bindmethod=simple 808> credentials=secret searchbase="cn=config" type=refreshAndPersist 809> retry="5 5 300 5" timeout=1 810> - 811> add: olcMirrorMode 812> olcMirrorMode: TRUE 813 814Now start up the Master and a consumer/s, also add the above LDIF to the first consumer, second consumer etc. It will then replicate {{B:cn=config}}. You now have N-Way Multimaster on the config database. 815 816We still have to replicate the actual data, not just the config, so add to the master (all active and configured consumers/masters will pull down this config, as they are all syncing). Also, replace all {{${}}} variables with whatever is applicable to your setup: 817 818> dn: olcDatabase={1}$BACKEND,cn=config 819> objectClass: olcDatabaseConfig 820> objectClass: olc${BACKEND}Config 821> olcDatabase: {1}$BACKEND 822> olcSuffix: $BASEDN 823> olcDbDirectory: ./db 824> olcRootDN: $MANAGERDN 825> olcRootPW: $PASSWD 826> olcLimits: dn.exact="$MANAGERDN" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited 827> olcSyncRepl: rid=004 provider=$URI1 binddn="$MANAGERDN" bindmethod=simple 828> credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly 829> interval=00:00:00:10 retry="5 5 300 5" timeout=1 830> olcSyncRepl: rid=005 provider=$URI2 binddn="$MANAGERDN" bindmethod=simple 831> credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly 832> interval=00:00:00:10 retry="5 5 300 5" timeout=1 833> olcSyncRepl: rid=006 provider=$URI3 binddn="$MANAGERDN" bindmethod=simple 834> credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly 835> interval=00:00:00:10 retry="5 5 300 5" timeout=1 836> olcMirrorMode: TRUE 837> 838> dn: olcOverlay=syncprov,olcDatabase={1}${BACKEND},cn=config 839> changetype: add 840> objectClass: olcOverlayConfig 841> objectClass: olcSyncProvConfig 842> olcOverlay: syncprov 843 844Note: All of your servers' clocks must be tightly synchronized using 845e.g. NTP {{http://www.ntp.org/}}, atomic clock, or some other reliable 846time reference. 847 848Note: As stated in {{slapd-config}}(5), URLs specified in {{olcSyncRepl}} 849directives are the URLs of the servers from which to replicate. These 850must exactly match the URLs {{slapd}} listens on ({{-h}} in {{SECT:Command-Line Options}}). 851Otherwise slapd may attempt to replicate from itself, causing a loop. 852 853H3: MirrorMode 854 855MirrorMode configuration is actually very easy. If you have ever setup a normal 856slapd syncrepl provider, then the only change is the following two directives: 857 858> mirrormode on 859> serverID 1 860 861Note: You need to make sure that the {{serverID}} of each mirror node is 862different and add it as a global configuration option. 863 864H4: Mirror Node Configuration 865 866The first step is to configure the syncrepl provider the same as in the 867{{SECT:Set up the provider slapd}} section. 868 869Here's a specific cut down example using {{SECT:LDAP Sync Replication}} in 870{{refreshAndPersist}} mode: 871 872MirrorMode node 1: 873 874> # Global section 875> serverID 1 876> # database section 877> 878> # syncrepl directive 879> syncrepl rid=001 880> provider=ldap://ldap-sid2.example.com 881> bindmethod=simple 882> binddn="cn=mirrormode,dc=example,dc=com" 883> credentials=mirrormode 884> searchbase="dc=example,dc=com" 885> schemachecking=on 886> type=refreshAndPersist 887> retry="60 +" 888> 889> mirrormode on 890 891MirrorMode node 2: 892 893> # Global section 894> serverID 2 895> # database section 896> 897> # syncrepl directive 898> syncrepl rid=001 899> provider=ldap://ldap-sid1.example.com 900> bindmethod=simple 901> binddn="cn=mirrormode,dc=example,dc=com" 902> credentials=mirrormode 903> searchbase="dc=example,dc=com" 904> schemachecking=on 905> type=refreshAndPersist 906> retry="60 +" 907> 908> mirrormode on 909 910It's simple really; each MirrorMode node is setup {{B:exactly}} the same, except 911that the {{serverID}} is unique, and each consumer is pointed to 912the other server. 913 914H5: Failover Configuration 915 916There are generally 2 choices for this; 1. Hardware proxies/load-balancing or 917dedicated proxy software, 2. using a Back-LDAP proxy as a syncrepl provider 918 919A typical enterprise example might be: 920 921!import "dual_dc.png"; align="center"; title="MirrorMode Enterprise Configuration" 922FT[align="Center"] Figure X.Y: MirrorMode in a Dual Data Center Configuration 923 924H5: Normal Consumer Configuration 925 926This is exactly the same as the {{SECT:Set up the consumer slapd}} section. It 927can either setup in normal {{SECT:syncrepl replication}} mode, or in 928{{SECT:delta-syncrepl replication}} mode. 929 930H4: MirrorMode Summary 931 932You will now have a directory architecture that provides all of the 933consistency guarantees of single-master replication, while also providing the 934high availability of multi-master replication. 935 936 937H3: Syncrepl Proxy 938 939!import "push-based-complete.png"; align="center"; title="Syncrepl Proxy Mode" 940FT[align="Center"] Figure X.Y: Replacing slurpd 941 942The following example is for a self-contained push-based replication solution: 943 944> ####################################################################### 945> # Standard OpenLDAP Master/Provider 946> ####################################################################### 947> 948> include /usr/local/etc/openldap/schema/core.schema 949> include /usr/local/etc/openldap/schema/cosine.schema 950> include /usr/local/etc/openldap/schema/nis.schema 951> include /usr/local/etc/openldap/schema/inetorgperson.schema 952> 953> include /usr/local/etc/openldap/slapd.acl 954> 955> modulepath /usr/local/libexec/openldap 956> moduleload back_hdb.la 957> moduleload syncprov.la 958> moduleload back_monitor.la 959> moduleload back_ldap.la 960> 961> pidfile /usr/local/var/slapd.pid 962> argsfile /usr/local/var/slapd.args 963> 964> loglevel sync stats 965> 966> database hdb 967> suffix "dc=suretecsystems,dc=com" 968> directory /usr/local/var/openldap-data 969> 970> checkpoint 1024 5 971> cachesize 10000 972> idlcachesize 10000 973> 974> index objectClass eq 975> # rest of indexes 976> index default sub 977> 978> rootdn "cn=admin,dc=suretecsystems,dc=com" 979> rootpw testing 980> 981> # syncprov specific indexing 982> index entryCSN eq 983> index entryUUID eq 984> 985> # syncrepl Provider for primary db 986> overlay syncprov 987> syncprov-checkpoint 1000 60 988> 989> # Let the replica DN have limitless searches 990> limits dn.exact="cn=replicator,dc=suretecsystems,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited 991> 992> database monitor 993> 994> database config 995> rootpw testing 996> 997> ############################################################################## 998> # Consumer Proxy that pulls in data via Syncrepl and pushes out via slapd-ldap 999> ############################################################################## 1000> 1001> database ldap 1002> # ignore conflicts with other databases, as we need to push out to same suffix 1003> hidden on 1004> suffix "dc=suretecsystems,dc=com" 1005> rootdn "cn=slapd-ldap" 1006> uri ldap://localhost:9012/ 1007> 1008> lastmod on 1009> 1010> # We don't need any access to this DSA 1011> restrict all 1012> 1013> acl-bind bindmethod=simple 1014> binddn="cn=replicator,dc=suretecsystems,dc=com" 1015> credentials=testing 1016> 1017> syncrepl rid=001 1018> provider=ldap://localhost:9011/ 1019> binddn="cn=replicator,dc=suretecsystems,dc=com" 1020> bindmethod=simple 1021> credentials=testing 1022> searchbase="dc=suretecsystems,dc=com" 1023> type=refreshAndPersist 1024> retry="5 5 300 5" 1025> 1026> overlay syncprov 1027 1028A replica configuration for this type of setup could be: 1029 1030> ####################################################################### 1031> # Standard OpenLDAP Slave without Syncrepl 1032> ####################################################################### 1033> 1034> include /usr/local/etc/openldap/schema/core.schema 1035> include /usr/local/etc/openldap/schema/cosine.schema 1036> include /usr/local/etc/openldap/schema/nis.schema 1037> include /usr/local/etc/openldap/schema/inetorgperson.schema 1038> 1039> include /usr/local/etc/openldap/slapd.acl 1040> 1041> modulepath /usr/local/libexec/openldap 1042> moduleload back_hdb.la 1043> moduleload syncprov.la 1044> moduleload back_monitor.la 1045> moduleload back_ldap.la 1046> 1047> pidfile /usr/local/var/slapd.pid 1048> argsfile /usr/local/var/slapd.args 1049> 1050> loglevel sync stats 1051> 1052> database hdb 1053> suffix "dc=suretecsystems,dc=com" 1054> directory /usr/local/var/openldap-slave/data 1055> 1056> checkpoint 1024 5 1057> cachesize 10000 1058> idlcachesize 10000 1059> 1060> index objectClass eq 1061> # rest of indexes 1062> index default sub 1063> 1064> rootdn "cn=admin,dc=suretecsystems,dc=com" 1065> rootpw testing 1066> 1067> # Let the replica DN have limitless searches 1068> limits dn.exact="cn=replicator,dc=suretecsystems,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited 1069> 1070> updatedn "cn=replicator,dc=suretecsystems,dc=com" 1071> 1072> # Refer updates to the master 1073> updateref ldap://localhost:9011 1074> 1075> database monitor 1076> 1077> database config 1078> rootpw testing 1079 1080You can see we use the {{updatedn}} directive here and example ACLs ({{F:usr/local/etc/openldap/slapd.acl}}) for this could be: 1081 1082> # Give the replica DN unlimited read access. This ACL may need to be 1083> # merged with other ACL statements. 1084> 1085> access to * 1086> by dn.base="cn=replicator,dc=suretecsystems,dc=com" write 1087> by * break 1088> 1089> access to dn.base="" 1090> by * read 1091> 1092> access to dn.base="cn=Subschema" 1093> by * read 1094> 1095> access to dn.subtree="cn=Monitor" 1096> by dn.exact="uid=admin,dc=suretecsystems,dc=com" write 1097> by users read 1098> by * none 1099> 1100> access to * 1101> by self write 1102> by * read 1103 1104In order to support more replicas, just add more {{database ldap}} sections and 1105increment the {{syncrepl rid}} number accordingly. 1106 1107Note: You must populate the Master and Slave directories with the same data, 1108unlike when using normal Syncrepl 1109 1110If you do not have access to modify the master directory configuration you can 1111configure a standalone ldap proxy, which might look like: 1112 1113!import "push-based-standalone.png"; align="center"; title="Syncrepl Standalone Proxy Mode" 1114FT[align="Center"] Figure X.Y: Replacing slurpd with a standalone version 1115 1116The following configuration is an example of a standalone LDAP Proxy: 1117 1118> include /usr/local/etc/openldap/schema/core.schema 1119> include /usr/local/etc/openldap/schema/cosine.schema 1120> include /usr/local/etc/openldap/schema/nis.schema 1121> include /usr/local/etc/openldap/schema/inetorgperson.schema 1122> 1123> include /usr/local/etc/openldap/slapd.acl 1124> 1125> modulepath /usr/local/libexec/openldap 1126> moduleload syncprov.la 1127> moduleload back_ldap.la 1128> 1129> ############################################################################## 1130> # Consumer Proxy that pulls in data via Syncrepl and pushes out via slapd-ldap 1131> ############################################################################## 1132> 1133> database ldap 1134> # ignore conflicts with other databases, as we need to push out to same suffix 1135> hidden on 1136> suffix "dc=suretecsystems,dc=com" 1137> rootdn "cn=slapd-ldap" 1138> uri ldap://localhost:9012/ 1139> 1140> lastmod on 1141> 1142> # We don't need any access to this DSA 1143> restrict all 1144> 1145> acl-bind bindmethod=simple 1146> binddn="cn=replicator,dc=suretecsystems,dc=com" 1147> credentials=testing 1148> 1149> syncrepl rid=001 1150> provider=ldap://localhost:9011/ 1151> binddn="cn=replicator,dc=suretecsystems,dc=com" 1152> bindmethod=simple 1153> credentials=testing 1154> searchbase="dc=suretecsystems,dc=com" 1155> type=refreshAndPersist 1156> retry="5 5 300 5" 1157> 1158> overlay syncprov 1159 1160As you can see, you can let your imagination go wild using Syncrepl and 1161{{slapd-ldap(8)}} tailoring your replication to fit your specific network 1162topology. 1163
