1# $OpenLDAP$ 2# Copyright 1999-2016 The OpenLDAP Foundation, All Rights Reserved. 3# COPYING RESTRICTIONS APPLY, see COPYRIGHT. 4 5H1: Replication 6 7Replicated directories are a fundamental requirement for delivering a 8resilient enterprise deployment. 9 10{{PRD:OpenLDAP}} has various configuration options for creating a replicated 11directory. In previous releases, replication was discussed in terms of 12a {{master}} server and some number of {{slave}} servers. A master 13accepted directory updates from other clients, and a slave only 14accepted updates from a (single) master. The replication structure 15was rigidly defined and any particular database could only fulfill 16a single role, either master or slave. 17 18As OpenLDAP now supports a wide variety of replication topologies, these 19terms have been deprecated in favor of {{provider}} and 20{{consumer}}: A provider replicates directory updates to consumers; 21consumers receive replication updates from providers. Unlike the 22rigidly defined master/slave relationships, provider/consumer roles 23are quite fluid: replication updates received in a consumer can be 24further propagated by that consumer to other servers, so a consumer 25can also act simultaneously as a provider. Also, a consumer need not 26be an actual LDAP server; it may be just an LDAP client. 27 28The following sections will describe the replication technology and 29discuss the various replication options that are available. 30 31H2: Replication Technology 32 33H3: LDAP Sync Replication 34 35The {{TERM:LDAP Sync}} Replication engine, {{TERM:syncrepl}} for 36short, is a consumer-side replication engine that enables the 37consumer {{TERM:LDAP}} server to maintain a shadow copy of a 38{{TERM:DIT}} fragment. A syncrepl engine resides at the consumer 39and executes as one of the {{slapd}}(8) threads. It creates and maintains a 40consumer replica by connecting to the replication provider to perform 41the initial DIT content load followed either by periodic content 42polling or by timely updates upon content changes. 43 44Syncrepl uses the LDAP Content Synchronization protocol (or LDAP Sync for 45short) as the replica synchronization protocol. LDAP Sync provides 46a stateful replication which supports both pull-based and push-based 47synchronization and does not mandate the use of a history store. 48In pull-based replication the consumer periodically 49polls the provider for updates. In push-based replication the consumer 50listens for updates that are sent by the provider in realtime. Since the 51protocol does not require a history store, the provider does not need to 52maintain any log of updates it has received (Note 53that the syncrepl engine is extensible and additional replication 54protocols may be supported in the future.). 55 56Syncrepl keeps track of the status of the replication content by 57maintaining and exchanging synchronization cookies. Because the 58syncrepl consumer and provider maintain their content status, the 59consumer can poll the provider content to perform incremental 60synchronization by asking for the entries required to make the 61consumer replica up-to-date with the provider content. Syncrepl 62also enables convenient management of replicas by maintaining replica 63status. The consumer replica can be constructed from a consumer-side 64or a provider-side backup at any synchronization status. Syncrepl 65can automatically resynchronize the consumer replica up-to-date 66with the current provider content. 67 68Syncrepl supports both pull-based and push-based synchronization. 69In its basic refreshOnly synchronization mode, the provider uses 70pull-based synchronization where the consumer servers need not be 71tracked and no history information is maintained. The information 72required for the provider to process periodic polling requests is 73contained in the synchronization cookie of the request itself. To 74optimize the pull-based synchronization, syncrepl utilizes the 75present phase of the LDAP Sync protocol as well as its delete phase, 76instead of falling back on frequent full reloads. To further optimize 77the pull-based synchronization, the provider can maintain a per-scope 78session log as a history store. In its refreshAndPersist mode of 79synchronization, the provider uses a push-based synchronization. 80The provider keeps track of the consumer servers that have requested 81a persistent search and sends them necessary updates as the provider 82replication content gets modified. 83 84With syncrepl, a consumer server can create a replica without 85changing the provider's configurations and without restarting the 86provider server, if the consumer server has appropriate access 87privileges for the DIT fragment to be replicated. The consumer 88server can stop the replication also without the need for provider-side 89changes and restart. 90 91Syncrepl supports partial, sparse, and fractional replications. The shadow 92DIT fragment is defined by a general search criteria consisting of 93base, scope, filter, and attribute list. The replica content is 94also subject to the access privileges of the bind identity of the 95syncrepl replication connection. 96 97 98H4: The LDAP Content Synchronization Protocol 99 100The LDAP Sync protocol allows a client to maintain a synchronized 101copy of a DIT fragment. The LDAP Sync operation is defined as a set 102of controls and other protocol elements which extend the LDAP search 103operation. This section introduces the LDAP Content Sync protocol 104only briefly. For more information, refer to {{REF:RFC4533}}. 105 106The LDAP Sync protocol supports both polling and listening for changes 107by defining two respective synchronization operations: 108{{refreshOnly}} and {{refreshAndPersist}}. Polling is implemented 109by the {{refreshOnly}} operation. The consumer 110polls the provider using an LDAP Search request with an LDAP Sync 111control attached. The consumer copy is synchronized 112to the provider copy at the time of polling using the information 113returned in the search. The provider finishes the 114search operation by returning {{SearchResultDone}} at the end of 115the search operation as in the normal search. Listening is 116implemented by the {{refreshAndPersist}} operation. As the name 117implies, it begins with a search, like refreshOnly. Instead of 118finishing the search after returning all entries currently matching 119the search criteria, the synchronization search remains persistent 120in the provider. Subsequent updates to the synchronization content 121in the provider cause additional entry updates to be sent to the 122consumer. 123 124The {{refreshOnly}} operation and the refresh stage of the 125{{refreshAndPersist}} operation can be performed with a present 126phase or a delete phase. 127 128In the present phase, the provider sends the consumer the entries updated 129within the search scope since the last synchronization. The provider 130sends all requested attributes, be they changed or not, of the updated 131entries. For each unchanged entry which remains in the scope, the 132provider sends a present message consisting only of the name of the 133entry and the synchronization control representing state present. 134The present message does not contain any attributes of the entry. 135After the consumer receives all update and present entries, it can 136reliably determine the new consumer copy by adding the entries added 137to the provider, by replacing the entries modified at the provider, and 138by deleting entries in the consumer copy which have not been updated 139nor specified as being present at the provider. 140 141The transmission of the updated entries in the delete phase is the 142same as in the present phase. The provider sends all the requested 143attributes of the entries updated within the search scope since the 144last synchronization to the consumer. In the delete phase, however, 145the provider sends a delete message for each entry deleted from the 146search scope, instead of sending present messages. The delete 147message consists only of the name of the entry and the synchronization 148control representing state delete. The new consumer copy can be 149determined by adding, modifying, and removing entries according to 150the synchronization control attached to the {{SearchResultEntry}} 151message. 152 153In the case that the LDAP Sync provider maintains a history store and 154can determine which entries are scoped out of the consumer copy since 155the last synchronization time, the provider can use the delete phase. 156If the provider does not maintain any history store, cannot determine 157the scoped-out entries from the history store, or the history store 158does not cover the outdated synchronization state of the consumer, 159the provider should use the present phase. The use of the present 160phase is much more efficient than a full content reload in terms 161of the synchronization traffic. To reduce the synchronization 162traffic further, the LDAP Sync protocol also provides several 163optimizations such as the transmission of the normalized {{EX:entryUUID}}s 164and the transmission of multiple {{EX:entryUUIDs}} in a single 165{{syncIdSet}} message. 166 167At the end of the {{refreshOnly}} synchronization, the provider sends 168a synchronization cookie to the consumer as a state indicator of the 169consumer copy after the synchronization is completed. The consumer 170will present the received cookie when it requests the next incremental 171synchronization to the provider. 172 173When {{refreshAndPersist}} synchronization is used, the provider sends 174a synchronization cookie at the end of the refresh stage by sending 175a Sync Info message with refreshDone=TRUE. It also sends a 176synchronization cookie by attaching it to {{SearchResultEntry}} 177messages generated in the persist stage of the synchronization search. During 178the persist stage, the provider can also send a Sync Info message 179containing the synchronization cookie at any time the provider wants 180to update the consumer-side state indicator. 181 182In the LDAP Sync protocol, entries are uniquely identified by the 183{{EX:entryUUID}} attribute value. It can function as a reliable 184identifier of the entry. The DN of the entry, on the other hand, 185can be changed over time and hence cannot be considered as the 186reliable identifier. The {{EX:entryUUID}} is attached to each 187{{SearchResultEntry}} or {{SearchResultReference}} as a part of the 188synchronization control. 189 190H4: Syncrepl Details 191 192The syncrepl engine utilizes both the {{refreshOnly}} and the 193{{refreshAndPersist}} operations of the LDAP Sync protocol. If a 194syncrepl specification is included in a database definition, 195{{slapd}}(8) launches a syncrepl engine as a {{slapd}}(8) thread 196and schedules its execution. If the {{refreshOnly}} operation is 197specified, the syncrepl engine will be rescheduled at the interval 198time after a synchronization operation is completed. If the 199{{refreshAndPersist}} operation is specified, the engine will remain 200active and process the persistent synchronization messages from the 201provider. 202 203The syncrepl engine utilizes both the present phase and the delete 204phase of the refresh synchronization. It is possible to configure 205a session log in the provider which stores the 206{{EX:entryUUID}}s of a finite number of entries deleted from a 207database. Multiple replicas share the same session log. The syncrepl 208engine uses the 209delete phase if the session log is present and the state of the 210consumer server is recent enough that no session log entries are 211truncated after the last synchronization of the client. The syncrepl 212engine uses the present phase if no session log is configured for 213the replication content or if the consumer replica is too outdated 214to be covered by the session log. The current design of the session 215log store is memory based, so the information contained in the 216session log is not persistent over multiple provider invocations. 217It is not currently supported to access the session log store by 218using LDAP operations. It is also not currently supported to impose 219access control to the session log. 220 221As a further optimization, even in the case the synchronization 222search is not associated with any session log, no entries will be 223transmitted to the consumer server when there has been no update 224in the replication context. 225 226The syncrepl engine, which is a consumer-side replication engine, 227can work with any backends. The LDAP Sync provider can be configured 228as an overlay on any backend, but works best with the {{back-bdb}} 229{{back-hdb}}, or {{back-mdb}} backends. 230 231The LDAP Sync provider maintains a {{EX:contextCSN}} for each 232database as the current synchronization state indicator of the 233provider content. It is the largest {{EX:entryCSN}} in the provider 234context such that no transactions for an entry having smaller 235{{EX:entryCSN}} value remains outstanding. The {{EX:contextCSN}} 236could not just be set to the largest issued {{EX:entryCSN}} because 237{{EX:entryCSN}} is obtained before a transaction starts and 238transactions are not committed in the issue order. 239 240The provider stores the {{EX:contextCSN}} of a context in the 241{{EX:contextCSN}} attribute of the context suffix entry. The attribute 242is not written to the database after every update operation though; 243instead it is maintained primarily in memory. At database start 244time the provider reads the last saved {{EX:contextCSN}} into memory 245and uses the in-memory copy exclusively thereafter. By default, 246changes to the {{EX:contextCSN}} as a result of database updates 247will not be written to the database until the server is cleanly 248shut down. A checkpoint facility exists to cause the {{EX:contextCSN}} to 249be written out more frequently if desired. 250 251Note that at startup time, if the provider is unable to read a 252{{EX:contextCSN}} from the suffix entry, it will scan the entire 253database to determine the value, and this scan may take quite a 254long time on a large database. When a {{EX:contextCSN}} value is 255read, the database will still be scanned for any {{EX:entryCSN}} 256values greater than it, to make sure the {{EX:contextCSN}} value 257truly reflects the greatest committed {{EX:entryCSN}} in the database. 258On databases which support inequality indexing, setting an eq index 259on the {{EX:entryCSN}} attribute and configuring {{contextCSN}} 260checkpoints will greatly speed up this scanning step. 261 262If no {{EX:contextCSN}} can be determined by reading and scanning 263the database, a new value will be generated. Also, if scanning the 264database yielded a greater {{EX:entryCSN}} than was previously 265recorded in the suffix entry's {{EX:contextCSN}} attribute, a 266checkpoint will be immediately written with the new value. 267 268The consumer also stores its replica state, which is the provider's 269{{EX:contextCSN}} received as a synchronization cookie, in the 270{{EX:contextCSN}} attribute of the suffix entry. The replica state 271maintained by a consumer server is used as the synchronization state 272indicator when it performs subsequent incremental synchronization 273with the provider server. It is also used as a provider-side 274synchronization state indicator when it functions as a secondary 275provider server in a cascading replication configuration. Since 276the consumer and provider state information are maintained in the 277same location within their respective databases, any consumer can 278be promoted to a provider (and vice versa) without any special 279actions. 280 281Because a general search filter can be used in the syncrepl 282specification, some entries in the context may be omitted from the 283synchronization content. The syncrepl engine creates a glue entry 284to fill in the holes in the replica context if any part of the 285replica content is subordinate to the holes. The glue entries will 286not be returned in the search result unless {{ManageDsaIT}} control 287is provided. 288 289Also as a consequence of the search filter used in the syncrepl 290specification, it is possible for a modification to remove an entry 291from the replication scope even though the entry has not been deleted 292on the provider. Logically the entry must be deleted on the consumer 293but in {{refreshOnly}} mode the provider cannot detect and propagate 294this change without the use of the session log on the provider. 295 296For configuration, please see the {{SECT:Syncrepl}} section. 297 298 299H2: Deployment Alternatives 300 301While the LDAP Sync specification only defines a narrow scope for replication, 302the OpenLDAP implementation is extremely flexible and supports a variety of 303operating modes to handle other scenarios not explicitly addressed in the spec. 304 305 306H3: Delta-syncrepl replication 307 308* Disadvantages of LDAP Sync replication: 309 310LDAP Sync replication is an object-based replication mechanism. 311When any attribute value in a replicated object is changed on the provider, 312each consumer fetches and processes the complete changed object, including 313{{B:both the changed and unchanged attribute values}} during replication. 314One advantage of this approach is that when multiple changes occur to 315a single object, the precise sequence of those changes need not be preserved; 316only the final state of the entry is significant. But this approach 317may have drawbacks when the usage pattern involves single changes to 318multiple objects. 319 320For example, suppose you have a database consisting of 102,400 objects of 1 KB 321each. Further, suppose you routinely run a batch job to change the value of 322a single two-byte attribute value that appears in each of the 102,400 objects 323on the master. Not counting LDAP and TCP/IP protocol overhead, each time you 324run this job each consumer will transfer and process {{B:100 MB}} of data to 325process {{B:200KB of changes!}} 326 32799.98% of the data that is transmitted and processed in a case like this will 328be redundant, since it represents values that did not change. This is a waste 329of valuable transmission and processing bandwidth and can cause an unacceptable 330replication backlog to develop. While this situation is extreme, it serves to 331demonstrate a very real problem that is encountered in some LDAP deployments. 332 333 334* Where Delta-syncrepl comes in: 335 336Delta-syncrepl, a changelog-based variant of syncrepl, is designed to address 337situations like the one described above. Delta-syncrepl works by maintaining a 338changelog of a selectable depth in a separate database on the provider. The replication consumer 339checks the changelog for the changes it needs and, as long as 340the changelog contains the needed changes, the consumer fetches the changes 341from the changelog and applies them to its database. If, however, a replica 342is too far out of sync (or completely empty), conventional syncrepl is used to 343bring it up to date and replication then switches back to the delta-syncrepl 344mode. 345 346Note: since the database state is stored in both the changelog DB and the 347main DB on the provider, it is important to backup/restore both the changelog 348DB and the main DB using slapcat/slapadd when restoring a DB or copying 349it to another machine. 350 351For configuration, please see the {{SECT:Delta-syncrepl}} section. 352 353 354H3: N-Way Multi-Master replication 355 356Multi-Master replication is a replication technique using Syncrepl to replicate 357data to multiple provider ("Master") Directory servers. 358 359H4: Valid Arguments for Multi-Master replication 360 361* If any provider fails, other providers will continue to accept updates 362* Avoids a single point of failure 363* Providers can be located in several physical sites i.e. distributed across 364the network/globe. 365* Good for Automatic failover/High Availability 366 367H4: Invalid Arguments for Multi-Master replication 368 369(These are often claimed to be advantages of Multi-Master replication but 370those claims are false): 371 372* It has {{B:NOTHING}} to do with load balancing 373* Providers {{B:must}} propagate writes to {{B:all}} the other servers, which 374means the network traffic and write load spreads across all 375of the servers the same as for single-master. 376* Server utilization and performance are at best identical for 377Multi-Master and Single-Master replication; at worst Single-Master is 378superior because indexing can be tuned differently to optimize for the 379different usage patterns between the provider and the consumers. 380 381H4: Arguments against Multi-Master replication 382 383* Breaks the data consistency guarantees of the directory model 384* {{URL:http://www.openldap.org/faq/data/cache/1240.html}} 385* If connectivity with a provider is lost because of a network partition, then 386"automatic failover" can just compound the problem 387* Typically, a particular machine cannot distinguish between losing contact 388 with a peer because that peer crashed, or because the network link has failed 389* If a network is partitioned and multiple clients start writing to each of the 390"masters" then reconciliation will be a pain; it may be best to simply deny 391writes to the clients that are partitioned from the single provider 392 393 394For configuration, please see the {{SECT:N-Way Multi-Master}} section below 395 396H3: MirrorMode replication 397 398MirrorMode is a hybrid configuration that provides all of the consistency 399guarantees of single-master replication, while also providing the high 400availability of multi-master. In MirrorMode two providers are set up to 401replicate from each other (as a multi-master configuration), but an 402external frontend is employed to direct all writes to only one of 403the two servers. The second provider will only be used for writes if 404the first provider crashes, at which point the frontend will switch to 405directing all writes to the second provider. When a crashed provider is 406repaired and restarted it will automatically catch up to any changes 407on the running provider and resync. 408 409H4: Arguments for MirrorMode 410 411* Provides a high-availability (HA) solution for directory writes (replicas handle reads) 412* As long as one provider is operational, writes can safely be accepted 413* Provider nodes replicate from each other, so they are always up to date and 414can be ready to take over (hot standby) 415* Syncrepl also allows the provider nodes to re-synchronize after any downtime 416 417 418H4: Arguments against MirrorMode 419 420* MirrorMode is not what is termed as a Multi-Master solution. This is because 421writes have to go to just one of the mirror nodes at a time 422* MirrorMode can be termed as Active-Active Hot-Standby, therefore an external 423server (slapd in proxy mode) or device (hardware load balancer) 424is needed to manage which provider is currently active 425* Backups are managed slightly differently 426- If backing up the Berkeley database itself and periodically backing up the 427transaction log files, then the same member of the mirror pair needs to be 428used to collect logfiles until the next database backup is taken 429 430For configuration, please see the {{SECT:MirrorMode}} section below 431 432 433H3: Syncrepl Proxy Mode 434 435While the LDAP Sync protocol supports both pull- and push-based replication, 436the push mode (refreshAndPersist) must still be initiated from the consumer 437before the provider can begin pushing changes. In some network configurations, 438particularly where firewalls restrict the direction in which connections 439can be made, a provider-initiated push mode may be needed. 440 441This mode can be configured with the aid of the LDAP Backend 442({{SECT: Backends}} and {{slapd-ldap(8)}}). Instead of running the 443syncrepl engine on the actual consumer, a slapd-ldap proxy is set up 444near (or collocated with) the provider that points to the consumer, 445and the syncrepl engine runs on the proxy. 446 447For configuration, please see the {{SECT:Syncrepl Proxy}} section. 448 449H4: Replacing Slurpd 450 451The old {{slurpd}} mechanism only operated in provider-initiated 452push mode. Slurpd replication was deprecated in favor of Syncrepl 453replication and has been completely removed from OpenLDAP 2.4. 454 455The slurpd daemon was the original replication mechanism inherited from 456UMich's LDAP and operated in push mode: the master pushed changes to the 457slaves. It was replaced for many reasons, in brief: 458 459 * It was not reliable 460 ** It was extremely sensitive to the ordering of records in the replog 461 ** It could easily go out of sync, at which point manual intervention was 462 required to resync the slave database with the master directory 463 ** It wasn't very tolerant of unavailable servers. If a slave went down 464 for a long time, the replog could grow to a size that was too large for 465 slurpd to process 466 * It only worked in push mode 467 * It required stopping and restarting the master to add new slaves 468 * It only supported single master replication 469 470Syncrepl has none of those weaknesses: 471 472 * Syncrepl is self-synchronizing; you can start with a consumer database 473 in any state from totally empty to fully synced and it will automatically 474 do the right thing to achieve and maintain synchronization 475 ** It is completely insensitive to the order in which changes occur 476 ** It guarantees convergence between the consumer and the provider 477 content without manual intervention 478 ** It can resynchronize regardless of how long a consumer stays out 479 of contact with the provider 480 * Syncrepl can operate in either direction 481 * Consumers can be added at any time without touching anything on the 482 provider 483 * Multi-master replication is supported 484 485 486H2: Configuring the different replication types 487 488H3: Syncrepl 489 490H4: Syncrepl configuration 491 492Because syncrepl is a consumer-side replication engine, the syncrepl 493specification is defined in {{slapd.conf}}(5) of the consumer 494server, not in the provider server's configuration file. The initial 495loading of the replica content can be performed either by starting 496the syncrepl engine with no synchronization cookie or by populating 497the consumer replica by loading an {{TERM:LDIF}} file dumped as a 498backup at the provider. 499 500When loading from a backup, it is not required to perform the initial 501loading from the up-to-date backup of the provider content. The 502syncrepl engine will automatically synchronize the initial consumer 503replica to the current provider content. As a result, it is not 504required to stop the provider server in order to avoid the replica 505inconsistency caused by the updates to the provider content during 506the content backup and loading process. 507 508When replicating a large scale directory, especially in a bandwidth 509constrained environment, it is advised to load the consumer replica 510from a backup instead of performing a full initial load using 511syncrepl. 512 513 514H4: Set up the provider slapd 515 516The provider is implemented as an overlay, so the overlay itself 517must first be configured in {{slapd.conf}}(5) before it can be 518used. The provider has only two configuration directives, for setting 519checkpoints on the {{EX:contextCSN}} and for configuring the session 520log. Because the LDAP Sync search is subject to access control, 521proper access control privileges should be set up for the replicated 522content. 523 524The {{EX:contextCSN}} checkpoint is configured by the 525 526> syncprov-checkpoint <ops> <minutes> 527 528directive. Checkpoints are only tested after successful write 529operations. If {{<ops>}} operations or more than {{<minutes>}} 530time has passed since the last checkpoint, a new checkpoint is 531performed. 532 533The session log is configured by the 534 535> syncprov-sessionlog <size> 536 537directive, where {{<size>}} is the maximum number of session log 538entries the session log can record. When a session log is configured, 539it is automatically used for all LDAP Sync searches within the 540database. 541 542Note that using the session log requires searching on the {{entryUUID}} 543attribute. Setting an eq index on this attribute will greatly benefit 544the performance of the session log on the provider. 545 546A more complete example of the {{slapd.conf}}(5) content is thus: 547 548> database mdb 549> maxsize 1073741824 550> suffix dc=Example,dc=com 551> rootdn dc=Example,dc=com 552> directory /var/ldap/db 553> index objectclass,entryCSN,entryUUID eq 554> 555> overlay syncprov 556> syncprov-checkpoint 100 10 557> syncprov-sessionlog 100 558 559 560H4: Set up the consumer slapd 561 562The syncrepl replication is specified in the database section of 563{{slapd.conf}}(5) for the replica context. The syncrepl engine 564is backend independent and the directive can be defined with any 565database type. 566 567> database mdb 568> maxsize 1073741824 569> suffix dc=Example,dc=com 570> rootdn dc=Example,dc=com 571> directory /var/ldap/db 572> index objectclass,entryCSN,entryUUID eq 573> 574> syncrepl rid=123 575> provider=ldap://provider.example.com:389 576> type=refreshOnly 577> interval=01:00:00:00 578> searchbase="dc=example,dc=com" 579> filter="(objectClass=organizationalPerson)" 580> scope=sub 581> attrs="cn,sn,ou,telephoneNumber,title,l" 582> schemachecking=off 583> bindmethod=simple 584> binddn="cn=syncuser,dc=example,dc=com" 585> credentials=secret 586 587In this example, the consumer will connect to the provider {{slapd}}(8) 588at port 389 of {{FILE:ldap://provider.example.com}} to perform a 589polling ({{refreshOnly}}) mode of synchronization once a day. It 590will bind as {{EX:cn=syncuser,dc=example,dc=com}} using simple 591authentication with password "secret". Note that the access control 592privilege of {{EX:cn=syncuser,dc=example,dc=com}} should be set 593appropriately in the provider to retrieve the desired replication 594content. Also the search limits must be high enough on the provider 595to allow the syncuser to retrieve a complete copy of the requested 596content. The consumer uses the rootdn to write to its database so 597it always has full permissions to write all content. 598 599The synchronization search in the above example will search for the 600entries whose objectClass is organizationalPerson in the entire 601subtree rooted at {{EX:dc=example,dc=com}}. The requested attributes 602are {{EX:cn}}, {{EX:sn}}, {{EX:ou}}, {{EX:telephoneNumber}}, 603{{EX:title}}, and {{EX:l}}. The schema checking is turned off, so 604that the consumer {{slapd}}(8) will not enforce entry schema 605checking when it processes updates from the provider {{slapd}}(8). 606 607For more detailed information on the syncrepl directive, see the 608{{SECT:syncrepl}} section of {{SECT:The slapd Configuration File}} 609chapter of this admin guide. 610 611 612H4: Start the provider and the consumer slapd 613 614The provider {{slapd}}(8) is not required to be restarted. 615{{contextCSN}} is automatically generated as needed: it might be 616originally contained in the {{TERM:LDIF}} file, generated by 617{{slapadd}} (8), generated upon changes in the context, or generated 618when the first LDAP Sync search arrives at the provider. If an 619LDIF file is being loaded which did not previously contain the 620{{contextCSN}}, the {{-w}} option should be used with {{slapadd}} 621(8) to cause it to be generated. This will allow the server to 622startup a little quicker the first time it runs. 623 624When starting a consumer {{slapd}}(8), it is possible to provide 625a synchronization cookie as the {{-c cookie}} command line option 626in order to start the synchronization from a specific state. The 627cookie is a comma separated list of name=value pairs. Currently 628supported syncrepl cookie fields are {{csn=<csn>}} and {{rid=<rid>}}. 629{{<csn>}} represents the current synchronization state of the 630consumer replica. {{<rid>}} identifies a consumer replica locally 631within the consumer server. It is used to relate the cookie to the 632syncrepl definition in {{slapd.conf}}(5) which has the matching 633replica identifier. The {{<rid>}} must have no more than 3 decimal 634digits. The command line cookie overrides the synchronization 635cookie stored in the consumer replica database. 636 637 638H3: Delta-syncrepl 639 640H4: Delta-syncrepl Provider configuration 641 642Setting up delta-syncrepl requires configuration changes on both the master and 643replica servers: 644 645> # Give the replica DN unlimited read access. This ACL needs to be 646> # merged with other ACL statements, and/or moved within the scope 647> # of a database. The "by * break" portion causes evaluation of 648> # subsequent rules. See slapd.access(5) for details. 649> access to * 650> by dn.base="cn=replicator,dc=symas,dc=com" read 651> by * break 652> 653> # Set the module path location 654> modulepath /opt/symas/lib/openldap 655> 656> # Load the hdb backend 657> moduleload back_hdb.la 658> 659> # Load the accesslog overlay 660> moduleload accesslog.la 661> 662> #Load the syncprov overlay 663> moduleload syncprov.la 664> 665> # Accesslog database definitions 666> database hdb 667> suffix cn=accesslog 668> directory /db/accesslog 669> rootdn cn=accesslog 670> index default eq 671> index entryCSN,objectClass,reqEnd,reqResult,reqStart 672> 673> overlay syncprov 674> syncprov-nopresent TRUE 675> syncprov-reloadhint TRUE 676> 677> # Let the replica DN have limitless searches 678> limits dn.exact="cn=replicator,dc=symas,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited 679> 680> # Primary database definitions 681> database hdb 682> suffix "dc=symas,dc=com" 683> rootdn "cn=manager,dc=symas,dc=com" 684> 685> ## Whatever other configuration options are desired 686> 687> # syncprov specific indexing 688> index entryCSN eq 689> index entryUUID eq 690> 691> # syncrepl Provider for primary db 692> overlay syncprov 693> syncprov-checkpoint 1000 60 694> 695> # accesslog overlay definitions for primary db 696> overlay accesslog 697> logdb cn=accesslog 698> logops writes 699> logsuccess TRUE 700> # scan the accesslog DB every day, and purge entries older than 7 days 701> logpurge 07+00:00 01+00:00 702> 703> # Let the replica DN have limitless searches 704> limits dn.exact="cn=replicator,dc=symas,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited 705 706For more information, always consult the relevant man pages ({{slapo-accesslog}}(5) and {{slapd.conf}}(5)) 707 708 709H4: Delta-syncrepl Consumer configuration 710 711> # Replica database configuration 712> database hdb 713> suffix "dc=symas,dc=com" 714> rootdn "cn=manager,dc=symas,dc=com" 715> 716> ## Whatever other configuration bits for the replica, like indexing 717> ## that you want 718> 719> # syncrepl specific indices 720> index entryUUID eq 721> 722> # syncrepl directives 723> syncrepl rid=0 724> provider=ldap://ldapmaster.symas.com:389 725> bindmethod=simple 726> binddn="cn=replicator,dc=symas,dc=com" 727> credentials=secret 728> searchbase="dc=symas,dc=com" 729> logbase="cn=accesslog" 730> logfilter="(&(objectClass=auditWriteObject)(reqResult=0))" 731> schemachecking=on 732> type=refreshAndPersist 733> retry="60 +" 734> syncdata=accesslog 735> 736> # Refer updates to the master 737> updateref ldap://ldapmaster.symas.com 738 739 740The above configuration assumes that you have a replicator identity defined 741in your database that can be used to bind to the provider. In addition, 742all of the databases (primary, replica, and the accesslog 743storage database) should also have properly tuned {{DB_CONFIG}} files that meet 744your needs. 745 746 747H3: N-Way Multi-Master 748 749For the following example we will be using 3 Master nodes. Keeping in line with 750{{B:test050-syncrepl-multimaster}} of the OpenLDAP test suite, we will be configuring 751{{slapd(8)}} via {{B:cn=config}} 752 753This sets up the config database: 754 755> dn: cn=config 756> objectClass: olcGlobal 757> cn: config 758> olcServerID: 1 759> 760> dn: olcDatabase={0}config,cn=config 761> objectClass: olcDatabaseConfig 762> olcDatabase: {0}config 763> olcRootPW: secret 764 765second and third servers will have a different olcServerID obviously: 766 767> dn: cn=config 768> objectClass: olcGlobal 769> cn: config 770> olcServerID: 2 771> 772> dn: olcDatabase={0}config,cn=config 773> objectClass: olcDatabaseConfig 774> olcDatabase: {0}config 775> olcRootPW: secret 776 777This sets up syncrepl as a provider (since these are all masters): 778 779> dn: cn=module,cn=config 780> objectClass: olcModuleList 781> cn: module 782> olcModulePath: /usr/local/libexec/openldap 783> olcModuleLoad: syncprov.la 784 785Now we setup the first Master Node (replace $URI1, $URI2 and $URI3 etc. with your actual ldap urls): 786 787> dn: cn=config 788> changetype: modify 789> replace: olcServerID 790> olcServerID: 1 $URI1 791> olcServerID: 2 $URI2 792> olcServerID: 3 $URI3 793> 794> dn: olcOverlay=syncprov,olcDatabase={0}config,cn=config 795> changetype: add 796> objectClass: olcOverlayConfig 797> objectClass: olcSyncProvConfig 798> olcOverlay: syncprov 799> 800> dn: olcDatabase={0}config,cn=config 801> changetype: modify 802> add: olcSyncRepl 803> olcSyncRepl: rid=001 provider=$URI1 binddn="cn=config" bindmethod=simple 804> credentials=secret searchbase="cn=config" type=refreshAndPersist 805> retry="5 5 300 5" timeout=1 806> olcSyncRepl: rid=002 provider=$URI2 binddn="cn=config" bindmethod=simple 807> credentials=secret searchbase="cn=config" type=refreshAndPersist 808> retry="5 5 300 5" timeout=1 809> olcSyncRepl: rid=003 provider=$URI3 binddn="cn=config" bindmethod=simple 810> credentials=secret searchbase="cn=config" type=refreshAndPersist 811> retry="5 5 300 5" timeout=1 812> - 813> add: olcMirrorMode 814> olcMirrorMode: TRUE 815 816Now start up the Master and a consumer/s, also add the above LDIF to the first consumer, second consumer etc. It will then replicate {{B:cn=config}}. You now have N-Way Multimaster on the config database. 817 818We still have to replicate the actual data, not just the config, so add to the master (all active and configured consumers/masters will pull down this config, as they are all syncing). Also, replace all {{${}}} variables with whatever is applicable to your setup: 819 820> dn: olcDatabase={1}$BACKEND,cn=config 821> objectClass: olcDatabaseConfig 822> objectClass: olc${BACKEND}Config 823> olcDatabase: {1}$BACKEND 824> olcSuffix: $BASEDN 825> olcDbDirectory: ./db 826> olcRootDN: $MANAGERDN 827> olcRootPW: $PASSWD 828> olcLimits: dn.exact="$MANAGERDN" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited 829> olcSyncRepl: rid=004 provider=$URI1 binddn="$MANAGERDN" bindmethod=simple 830> credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly 831> interval=00:00:00:10 retry="5 5 300 5" timeout=1 832> olcSyncRepl: rid=005 provider=$URI2 binddn="$MANAGERDN" bindmethod=simple 833> credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly 834> interval=00:00:00:10 retry="5 5 300 5" timeout=1 835> olcSyncRepl: rid=006 provider=$URI3 binddn="$MANAGERDN" bindmethod=simple 836> credentials=$PASSWD searchbase="$BASEDN" type=refreshOnly 837> interval=00:00:00:10 retry="5 5 300 5" timeout=1 838> olcMirrorMode: TRUE 839> 840> dn: olcOverlay=syncprov,olcDatabase={1}${BACKEND},cn=config 841> changetype: add 842> objectClass: olcOverlayConfig 843> objectClass: olcSyncProvConfig 844> olcOverlay: syncprov 845 846Note: All of your servers' clocks must be tightly synchronized using 847e.g. NTP {{http://www.ntp.org/}}, atomic clock, or some other reliable 848time reference. 849 850Note: As stated in {{slapd-config}}(5), URLs specified in {{olcSyncRepl}} 851directives are the URLs of the servers from which to replicate. These 852must exactly match the URLs {{slapd}} listens on ({{-h}} in {{SECT:Command-Line Options}}). 853Otherwise slapd may attempt to replicate from itself, causing a loop. 854 855H3: MirrorMode 856 857MirrorMode configuration is actually very easy. If you have ever setup a normal 858slapd syncrepl provider, then the only change is the following two directives: 859 860> mirrormode on 861> serverID 1 862 863Note: You need to make sure that the {{serverID}} of each mirror node is 864different and add it as a global configuration option. 865 866H4: Mirror Node Configuration 867 868The first step is to configure the syncrepl provider the same as in the 869{{SECT:Set up the provider slapd}} section. 870 871Here's a specific cut down example using {{SECT:LDAP Sync Replication}} in 872{{refreshAndPersist}} mode: 873 874MirrorMode node 1: 875 876> # Global section 877> serverID 1 878> # database section 879> 880> # syncrepl directive 881> syncrepl rid=001 882> provider=ldap://ldap-sid2.example.com 883> bindmethod=simple 884> binddn="cn=mirrormode,dc=example,dc=com" 885> credentials=mirrormode 886> searchbase="dc=example,dc=com" 887> schemachecking=on 888> type=refreshAndPersist 889> retry="60 +" 890> 891> mirrormode on 892 893MirrorMode node 2: 894 895> # Global section 896> serverID 2 897> # database section 898> 899> # syncrepl directive 900> syncrepl rid=001 901> provider=ldap://ldap-sid1.example.com 902> bindmethod=simple 903> binddn="cn=mirrormode,dc=example,dc=com" 904> credentials=mirrormode 905> searchbase="dc=example,dc=com" 906> schemachecking=on 907> type=refreshAndPersist 908> retry="60 +" 909> 910> mirrormode on 911 912It's simple really; each MirrorMode node is setup {{B:exactly}} the same, except 913that the {{serverID}} is unique, and each consumer is pointed to 914the other server. 915 916H5: Failover Configuration 917 918There are generally 2 choices for this; 1. Hardware proxies/load-balancing or 919dedicated proxy software, 2. using a Back-LDAP proxy as a syncrepl provider 920 921A typical enterprise example might be: 922 923!import "dual_dc.png"; align="center"; title="MirrorMode Enterprise Configuration" 924FT[align="Center"] Figure X.Y: MirrorMode in a Dual Data Center Configuration 925 926H5: Normal Consumer Configuration 927 928This is exactly the same as the {{SECT:Set up the consumer slapd}} section. It 929can either setup in normal {{SECT:syncrepl replication}} mode, or in 930{{SECT:delta-syncrepl replication}} mode. 931 932H4: MirrorMode Summary 933 934You will now have a directory architecture that provides all of the 935consistency guarantees of single-master replication, while also providing the 936high availability of multi-master replication. 937 938 939H3: Syncrepl Proxy 940 941!import "push-based-complete.png"; align="center"; title="Syncrepl Proxy Mode" 942FT[align="Center"] Figure X.Y: Replacing slurpd 943 944The following example is for a self-contained push-based replication solution: 945 946> ####################################################################### 947> # Standard OpenLDAP Master/Provider 948> ####################################################################### 949> 950> include /usr/local/etc/openldap/schema/core.schema 951> include /usr/local/etc/openldap/schema/cosine.schema 952> include /usr/local/etc/openldap/schema/nis.schema 953> include /usr/local/etc/openldap/schema/inetorgperson.schema 954> 955> include /usr/local/etc/openldap/slapd.acl 956> 957> modulepath /usr/local/libexec/openldap 958> moduleload back_hdb.la 959> moduleload syncprov.la 960> moduleload back_monitor.la 961> moduleload back_ldap.la 962> 963> pidfile /usr/local/var/slapd.pid 964> argsfile /usr/local/var/slapd.args 965> 966> loglevel sync stats 967> 968> database hdb 969> suffix "dc=suretecsystems,dc=com" 970> directory /usr/local/var/openldap-data 971> 972> checkpoint 1024 5 973> cachesize 10000 974> idlcachesize 10000 975> 976> index objectClass eq 977> # rest of indexes 978> index default sub 979> 980> rootdn "cn=admin,dc=suretecsystems,dc=com" 981> rootpw testing 982> 983> # syncprov specific indexing 984> index entryCSN eq 985> index entryUUID eq 986> 987> # syncrepl Provider for primary db 988> overlay syncprov 989> syncprov-checkpoint 1000 60 990> 991> # Let the replica DN have limitless searches 992> limits dn.exact="cn=replicator,dc=suretecsystems,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited 993> 994> database monitor 995> 996> database config 997> rootpw testing 998> 999> ############################################################################## 1000> # Consumer Proxy that pulls in data via Syncrepl and pushes out via slapd-ldap 1001> ############################################################################## 1002> 1003> database ldap 1004> # ignore conflicts with other databases, as we need to push out to same suffix 1005> hidden on 1006> suffix "dc=suretecsystems,dc=com" 1007> rootdn "cn=slapd-ldap" 1008> uri ldap://localhost:9012/ 1009> 1010> lastmod on 1011> 1012> # We don't need any access to this DSA 1013> restrict all 1014> 1015> acl-bind bindmethod=simple 1016> binddn="cn=replicator,dc=suretecsystems,dc=com" 1017> credentials=testing 1018> 1019> syncrepl rid=001 1020> provider=ldap://localhost:9011/ 1021> binddn="cn=replicator,dc=suretecsystems,dc=com" 1022> bindmethod=simple 1023> credentials=testing 1024> searchbase="dc=suretecsystems,dc=com" 1025> type=refreshAndPersist 1026> retry="5 5 300 5" 1027> 1028> overlay syncprov 1029 1030A replica configuration for this type of setup could be: 1031 1032> ####################################################################### 1033> # Standard OpenLDAP Slave without Syncrepl 1034> ####################################################################### 1035> 1036> include /usr/local/etc/openldap/schema/core.schema 1037> include /usr/local/etc/openldap/schema/cosine.schema 1038> include /usr/local/etc/openldap/schema/nis.schema 1039> include /usr/local/etc/openldap/schema/inetorgperson.schema 1040> 1041> include /usr/local/etc/openldap/slapd.acl 1042> 1043> modulepath /usr/local/libexec/openldap 1044> moduleload back_hdb.la 1045> moduleload syncprov.la 1046> moduleload back_monitor.la 1047> moduleload back_ldap.la 1048> 1049> pidfile /usr/local/var/slapd.pid 1050> argsfile /usr/local/var/slapd.args 1051> 1052> loglevel sync stats 1053> 1054> database hdb 1055> suffix "dc=suretecsystems,dc=com" 1056> directory /usr/local/var/openldap-slave/data 1057> 1058> checkpoint 1024 5 1059> cachesize 10000 1060> idlcachesize 10000 1061> 1062> index objectClass eq 1063> # rest of indexes 1064> index default sub 1065> 1066> rootdn "cn=admin,dc=suretecsystems,dc=com" 1067> rootpw testing 1068> 1069> # Let the replica DN have limitless searches 1070> limits dn.exact="cn=replicator,dc=suretecsystems,dc=com" time.soft=unlimited time.hard=unlimited size.soft=unlimited size.hard=unlimited 1071> 1072> updatedn "cn=replicator,dc=suretecsystems,dc=com" 1073> 1074> # Refer updates to the master 1075> updateref ldap://localhost:9011 1076> 1077> database monitor 1078> 1079> database config 1080> rootpw testing 1081 1082You can see we use the {{updatedn}} directive here and example ACLs ({{F:usr/local/etc/openldap/slapd.acl}}) for this could be: 1083 1084> # Give the replica DN unlimited read access. This ACL may need to be 1085> # merged with other ACL statements. 1086> 1087> access to * 1088> by dn.base="cn=replicator,dc=suretecsystems,dc=com" write 1089> by * break 1090> 1091> access to dn.base="" 1092> by * read 1093> 1094> access to dn.base="cn=Subschema" 1095> by * read 1096> 1097> access to dn.subtree="cn=Monitor" 1098> by dn.exact="uid=admin,dc=suretecsystems,dc=com" write 1099> by users read 1100> by * none 1101> 1102> access to * 1103> by self write 1104> by * read 1105 1106In order to support more replicas, just add more {{database ldap}} sections and 1107increment the {{syncrepl rid}} number accordingly. 1108 1109Note: You must populate the Master and Slave directories with the same data, 1110unlike when using normal Syncrepl 1111 1112If you do not have access to modify the master directory configuration you can 1113configure a standalone ldap proxy, which might look like: 1114 1115!import "push-based-standalone.png"; align="center"; title="Syncrepl Standalone Proxy Mode" 1116FT[align="Center"] Figure X.Y: Replacing slurpd with a standalone version 1117 1118The following configuration is an example of a standalone LDAP Proxy: 1119 1120> include /usr/local/etc/openldap/schema/core.schema 1121> include /usr/local/etc/openldap/schema/cosine.schema 1122> include /usr/local/etc/openldap/schema/nis.schema 1123> include /usr/local/etc/openldap/schema/inetorgperson.schema 1124> 1125> include /usr/local/etc/openldap/slapd.acl 1126> 1127> modulepath /usr/local/libexec/openldap 1128> moduleload syncprov.la 1129> moduleload back_ldap.la 1130> 1131> ############################################################################## 1132> # Consumer Proxy that pulls in data via Syncrepl and pushes out via slapd-ldap 1133> ############################################################################## 1134> 1135> database ldap 1136> # ignore conflicts with other databases, as we need to push out to same suffix 1137> hidden on 1138> suffix "dc=suretecsystems,dc=com" 1139> rootdn "cn=slapd-ldap" 1140> uri ldap://localhost:9012/ 1141> 1142> lastmod on 1143> 1144> # We don't need any access to this DSA 1145> restrict all 1146> 1147> acl-bind bindmethod=simple 1148> binddn="cn=replicator,dc=suretecsystems,dc=com" 1149> credentials=testing 1150> 1151> syncrepl rid=001 1152> provider=ldap://localhost:9011/ 1153> binddn="cn=replicator,dc=suretecsystems,dc=com" 1154> bindmethod=simple 1155> credentials=testing 1156> searchbase="dc=suretecsystems,dc=com" 1157> type=refreshAndPersist 1158> retry="5 5 300 5" 1159> 1160> overlay syncprov 1161 1162As you can see, you can let your imagination go wild using Syncrepl and 1163{{slapd-ldap(8)}} tailoring your replication to fit your specific network 1164topology. 1165
