1slapd internal APIs 2 3Introduction 4 5Frontend, backend, database, callback, overlay - what does it all mean? 6 7The "frontend" refers to all the code that deals with the actual interaction 8with an LDAP client. This includes the code to read requests from the network 9and parse them into C data structures, all of the session management, and the 10formatting of responses for transmission onto the network. It also includes the 11access control engine and other features that are generic to LDAP processing, 12features which are not dependent on a particular database implementation. 13Because the frontend serves as the framework that ties everything together, 14it should not change much over time. 15 16The terms "backend" and "database" have historically been used interchangeably 17and/or in combination as if they are the same thing, but the code has a clear 18distinction between the two. A "backend" is a type of module, and a "database" 19is an instance of a backend type. Together they work with the frontend to 20manage the actual data that are operated on by LDAP requests. Originally the 21backend interface was relatively compact, with individual functions 22corresponding to each LDAP operation type, plus functions for init, config, and 23shutdown. The number of entry points has grown to allow greater flexibility, 24but the concept is much the same as before. 25 26The language here can get a bit confusing. A backend in slapd is embodied in a 27BackendInfo data structure, and a database is held in a BackendDB structure. 28Originally it was all just a single Backend data structure, but things have 29grown and the concept was split into these two parts. The idea behind the 30distinct BackendInfo is to allow for any initialization and configuration that 31may be needed by every instance of a type of database, as opposed to items that 32are specific to just one instance. For example, you might have a database 33library that requires an initialization routine to be called exactly once at 34program startup. Then there may be a "open" function that must be called once 35for each database instance. The BackendInfo.bi_open function provides the 36one-time startup, while the BackendInfo.bi_db_open function provides the 37per-database startup. The main feature of the BackendInfo structure is its 38table of entry points for all of the database functions that it implements. 39There's also a bi_private pointer that can be used to carry any configuration 40state needed by the backend. (Note that this is state that applies to the 41backend type, and thus to all database instances of the backend as well.) The 42BackendDB structure carries all of the per-instance state for a backend 43database. This includes the database suffix, ACLs, flags, various DNs, etc. It 44also has a pointer to its BackendInfo, and a be_private pointer for use by the 45particular backend instance. In practice, the per-type features are seldom 46used, and all of the work is done in the per-instance data structures. 47 48Ordinarily an LDAP request is received by the slapd frontend, parsed into a 49request structure, and then passed to the backend for processing. The backend 50may call various utility functions in the frontend to assist in processing, and 51then it eventually calls some send_ldap_result function in the frontend to send 52results back to the client. The processing flow is pretty rigidly defined; even 53though slapd is capable of dynamically loading new code modules, it was 54difficult to add extensions that changed the basic protocol operations. If you 55wanted to extend the server with special behaviors you would need to modify the 56frontend or the backend or both, and generally you would need to write an 57entire new backend to get some set of special features working. With OpenLDAP 582.1 we added the notion of a callback, which can intercept the results sent 59from a backend before they are sent to a client. Using callbacks makes it 60possible to modify the results if desired, or to simply discard the results 61instead of sending them to any client. This callback feature is used 62extensively in the SASL support to perform internal searches of slapd databases 63when mapping authentication IDs into regular DNs. The callback mechanism is 64also the basis of backglue, which allows separate databases to be searched as 65if they were a single naming context. 66 67Very often, one needs to add just a tiny feature onto an otherwise "normal" 68database. The usual way to achieve this was to use a programmable backend (like 69back-perl) to preprocess various requests and then forward them back into slapd 70to be handled by the real database. While this technique works, it is fairly 71inefficient because it involves many transitions from network to slapd and back 72again. The overlay concept introduced in OpenLDAP 2.2 allows code to be 73inserted between the slapd frontend and any backend, so that incoming requests 74can be intercepted before reaching the backend database. (There is also a SLAPI 75plugin framework in OpenLDAP 2.2; it offers a lot of flexibility as well but is 76not discussed here.) The overlay framework also uses the callback mechanism, so 77outgoing results can also be intercepted by external code. All of this could 78get unwieldy if a lot of overlays were being used, but there was also another 79significant API change in OpenLDAP 2.2 to streamline internal processing. (See 80the document "Refactoring the slapd ABI"...) 81 82OK, enough generalities... You should probably have a copy of slap.h in front 83of you to continue here. 84 85What is an overlay? The structure defining it includes a BackendInfo structure 86plus a few additional fields. It gets inserted into the usual frontend->backend 87call chain by replacing the BackendDB's BackendInfo pointer with its own. The 88framework to accomplish this is in backover.c. For a given backend, the 89BackendInfo will point to a slap_overinfo structure. The slap_overinfo has a 90BackendInfo that points to all of the overlay framework's entry points. It also 91holds a copy of the original BackendInfo pointer, and a linked list of 92slap_overinst structures. There is one slap_overinst per configured overlay, 93and the set of overlays configured on a backend are treated like a stack; i.e., 94the last one configured is at the top of the stack, and it executes first. 95 96Continuing with the stack notion - a request enters the frontend, is directed 97to a backend by select_backend, and then intercepted by the top of the overlay 98stack. This first overlay may do something with the request, and then return 99SLAP_CB_CONTINUE, which will then cause processing to fall into the next 100overlay, and so on down the stack until finally the request is handed to the 101actual backend database. Likewise, when the database finishes processing and 102sends a result, the overlay callback intercepts this and the topmost overlay 103gets to process the result. If it returns SLAP_CB_CONTINUE then processing will 104continue in the next overlay, and then any other callbacks, then finally the 105result reaches the frontend for sending back to the client. At any step along 106the way, a module may choose to fully process the request or result and not 107allow it to propagate any further down the stack. Whenever a module returns 108anything other than SLAP_CB_CONTINUE the processing stops. 109 110An overlay can call most frontend functions without any special consideration. 111However, if a call is going to result in any backend code being invoked, then 112the backend environment must be correct. During a normal backend invocation, 113op->o_bd points to the BackendDB structure for the backend, and 114op->o_bd->bd_info points to the BackendInfo for the backend. All of the 115information a specific backend instance needs is in op->o_bd->be_private and 116all of its entry points are in the BackendInfo structure. When overlays are in 117use on a backend, op->o_bd->bd_info points to the BackendInfo (actually a 118slap_overinfo) that contains the overlay framework. When a particular overlay 119instance is executing, op->o_bd points to a copy of the original op->o_bd, and 120op->o_bd->bd_info points to a slap_overinst which carries the information about 121the current overlay. The slap_overinst contains an on_private pointer which can 122be used to carry any configuration or state information the overlay needs. The 123normal way to invoke a backend function is through the op->o_bd->bd_info table 124of entry points, but obviously this must be set to the backend's original 125BackendInfo in order to get to the right function. 126 127There are two approaches here. The slap_overinst also contains a on_info field 128that points to the top slap_overinfo that wraps the current backend. The 129simplest thing is for the overlay to set op->o_bd->bd_info to this on_info 130value before invoking a backend function. This will cause processing of that 131particular operation to begin at the top of the overlay stack, so all the other 132overlays on the backend will also get a chance to handle this internal request. 133The other possibility is to invoke the underlying backend directly, bypassing 134the rest of the overlays, by calling through on_info->oi_orig. You should be 135careful in choosing this approach, since it precludes other overlays from doing 136their jobs. 137 138One of the more interesting uses for an overlay is to attach two (or more) 139different database backends into a single execution stack. Assuming that the 140basic frontend-managed information (suffix, rootdn, ACLs, etc.) will be the 141same for all of the backends, the only thing the overlay needs to maintain is a 142be_private and bd_info pointer for the added backends. The chain and proxycache 143overlays are two complementary examples of this usage. The chain overlay 144attaches a back-ldap backend to a local database backend, and allows referrals 145to remote servers generated by the database to be processed by slapd instead of 146being returned to the client. The proxycache overlay attaches a local database 147to a back-ldap (or back-meta) backend and allows search results from remote 148servers to be cached locally. In both cases the overlays must provide a bit of 149glue to swap in the appropriate be_private and bd_info pointers before invoking 150the attached backend, which can then be invoked as usual. 151 152Note on overlay initialization/destruction: you should allocate storage for 153config info in the _db_init handler, and free this storage in the _db_destroy 154handler. You must not free it in the _db_close handler because a module may 155be opened/closed multiple times in a running slapd when using dynamic 156configuration and the config info must remain intact. 157 158--- 159