1Part 1: Kea Migration Assistant support 2======================================= 3 4Files: 5------ 6 - data.h (tailq list and element type declarations) 7 - data.c (element type code) 8 - keama.h (DHCP declarations) 9 - keama.c (main() code) 10 - json.c (JSON parser) 11 - option.c (option tables and code) 12 - keama.8 (man page) 13 14The code heavily uses tailq lists, i.e. doubled linked lists with 15a pointer to the last (tail) element. 16 17The element structure mimics the Kea Element class with a few differences: 18 - no smart pointers 19 - extra fields to handle declaration kind, skip and comments 20 - maps are implemented as lists with an extra key field so the order 21 of insertion is kept and duplicates are possible 22 - strings are length + content (vs C strings) 23 24There is no attempt to avoid memory leaks. 25 26The skip flag is printed as '//' at the beginning of lines. It is set 27when something cannot be converted and the issue counter (returned 28by the keama command) incremented. 29 30Part 2: ISC DHCP lexer organization 31=================================== 32 33Files: 34----- 35 - dhctoken.h (from includes, enum dhcp_token definition) 36 - conflex.c (from common, lexical analyzer code) 37 38Tokens (dhcp_token enum): characters are set to their ASCII value, 39 others are >= 256 without real organization (e.g. END_OF_FILE is 607). 40 41The state is in a parse structure named "cfile". There is one per file 42and a few routine save it in order to do a backtrack on a larger 43set than the usual lookahead. 44The largest function is intern() which recognizes keywords with 45a switch on the first character and a tree of if strcasecmp's. 46 47Standard routines: 48----------------- 49enum dhcp_token 50next_token(const char **rval, unsigned *rlen, struct parse *cfile); 51 52and 53 54enum dhcp_token 55peek_token(const char **rval, unsigned *rlen, struct parse *cfile); 56 57rval: if not null the content of the token is put in it 58rlen: if not null the length of the token is put in it 59cfile: lexer context 60return: the integer value of the token 61 62Changes: 63------- 64 65Added LBRACKET '[' and RBRACKET ']' tokens for JSON parser 66(switch on dhcp_token type). 67 68Added comments to collect ISC DHCP # comments, element stack to follow 69declaration hierarchy, and issue counter to struct parse. 70 71Moved the parse_warn (renamed into parse_error and made fatal) routine 72from conflex.c to keama.c 73 74Part 3: ISC DHCP parser organization 75==================================== 76 77Files: 78----- 79 - confparse.c (from server) 80 for the server in parse_statement()) 81 - parse.c (from common) 82 834 classes: parameters, declarations, executable statements and expressions. 84 85the original code parses config and lease files, I kept only the first 86at the exception of parse_binding_value(). 87 88entry point 89 | 90 V 91conf_file_parse 92 | 93 V 94conf_file_subparse <- read_conf_file (for include) 95 until END_OF_FILE call 96 | 97 V 98parse_statement 99 parse parameters and declarations 100 switch on token and call parse_xxx_declaration routines 101 on default or DHCPv6 token in DHCPv4 mode call parse_executable_statement 102 and put the result under the "statement" key 103 | 104 V 105parse_executable_statement 106 107According to comments the grammar is: 108 109 conf-file :== parameters declarations END_OF_FILE 110 parameters :== <nil> | parameter | parameters parameter 111 declarations :== <nil> | declaration | declarations declaration 112 113 statement :== parameter | declaration 114 115 parameter :== DEFAULT_LEASE_TIME lease_time 116 | MAX_LEASE_TIME lease_time 117 | DYNAMIC_BOOTP_LEASE_CUTOFF date 118 | DYNAMIC_BOOTP_LEASE_LENGTH lease_time 119 | BOOT_UNKNOWN_CLIENTS boolean 120 | ONE_LEASE_PER_CLIENT boolean 121 | GET_LEASE_HOSTNAMES boolean 122 | USE_HOST_DECL_NAME boolean 123 | NEXT_SERVER ip-addr-or-hostname SEMI 124 | option_parameter 125 | SERVER-IDENTIFIER ip-addr-or-hostname SEMI 126 | FILENAME string-parameter 127 | SERVER_NAME string-parameter 128 | hardware-parameter 129 | fixed-address-parameter 130 | ALLOW allow-deny-keyword 131 | DENY allow-deny-keyword 132 | USE_LEASE_ADDR_FOR_DEFAULT_ROUTE boolean 133 | AUTHORITATIVE 134 | NOT AUTHORITATIVE 135 136 declaration :== host-declaration 137 | group-declaration 138 | shared-network-declaration 139 | subnet-declaration 140 | VENDOR_CLASS class-declaration 141 | USER_CLASS class-declaration 142 | RANGE address-range-declaration 143 144Typically declarations use { } and are associated with a group 145(changed to a type) in ROOT_GROUP (global), HOST_DECL, SHARED_NET_DECL, 146SUBNET_DECL, CLASS_DECL, GROUP_DECL and POOL_DECL. 147 148ROOT: parent = TOPLEVEL, children = everythig but not POOL 149HOST: parent = ROOT, GROUP, warn on SHARED or SUBNET, children = none 150SHARED_NET: parent = ROOT, GROUP, children = HOST (warn), SUBNET, POOL4 151SUBNET: parent = ROOT, GROUP, SHARED, children = HOST (warn), POOL 152CLASS: parent = ROOT, GROUP, children = none 153GROUP: parent = ROOT, SHARED, children = anything but not POOL 154POOL: parent = SHARED4, SUBNET, warn on others, children = none 155 156isc_boolean_t 157parse_statement(struct parse *cfile, int type, isc_boolean_t declaration); 158 159cfile: parser context 160type: declaration type 161declaration and return: declaration or parameter 162 163On the common side: 164 165 executable-statements :== executable-statement executable-statements | 166 executable-statement 167 168 executable-statement :== 169 IF if-statement | 170 ADD class-name SEMI | 171 BREAK SEMI | 172 OPTION option-parameter SEMI | 173 SUPERSEDE option-parameter SEMI | 174 PREPEND option-parameter SEMI | 175 APPEND option-parameter SEMI 176 177isc_boolean_t 178parse_executable_statement(struct element *result, 179 struct parse *cfile, isc_boolean_t *lose, 180 enum expression_context case_context, 181 isc_boolean_t direct); 182 183result: map element where to put the statement 184cfile: parser context 185lose: set to ISC_TRUE on failure 186case_context: expression context 187direct: called directly by parse_statement so can execute config statements 188return: success 189 190parse_executable_statement 191 switch on keywords (far more than in the comments) 192 on default with an identifier try a config option, on number or name 193 call parse_expression for a function call 194 | 195 V 196parse_expression 197 198expressions are divided into boolean, data (string) and numeric expressions 199 200 boolean_expression :== CHECK STRING | 201 NOT boolean-expression | 202 data-expression EQUAL data-expression | 203 data-expression BANG EQUAL data-expression | 204 data-expression REGEX_MATCH data-expression | 205 boolean-expression AND boolean-expression | 206 boolean-expression OR boolean-expression 207 EXISTS OPTION-NAME 208 209 data_expression :== SUBSTRING LPAREN data-expression COMMA 210 numeric-expression COMMA 211 numeric-expression RPAREN | 212 CONCAT LPAREN data-expression COMMA 213 data-expression RPAREN 214 SUFFIX LPAREN data_expression COMMA 215 numeric-expression RPAREN | 216 LCASE LPAREN data_expression RPAREN | 217 UCASE LPAREN data_expression RPAREN | 218 OPTION option_name | 219 HARDWARE | 220 PACKET LPAREN numeric-expression COMMA 221 numeric-expression RPAREN | 222 V6RELAY LPAREN numeric-expression COMMA 223 data-expression RPAREN | 224 STRING | 225 colon_separated_hex_list 226 227 numeric-expression :== EXTRACT_INT LPAREN data-expression 228 COMMA number RPAREN | 229 NUMBER 230 231parse_boolean_expression, parse_data_expression and parse_numeric_expression 232calls parse_expression and check its result 233 234parse_expression itself is divided into parse_non_binary and internal 235handling of binary operators 236 237isc_boolean_t 238parse_non_binary(struct element *expr, struct parse *cfile, 239 isc_boolean_t *lose, enum expression_context context) 240 241isc_boolean_t 242parse_expression(struct element *expr, struct parse *cfile, 243 isc_boolean_t *lose, enum expression_context context, 244 struct element *lhs, enum expr_op binop) 245 246expr: map element where to put the result 247cfile: parser context 248lose: set to ISC_TRUE on failure 249context: expression context 250lhs: NULL or left hand side 251binop: expr_none or binary operation 252return: success 253 254parse_non_binary 255 switch on unary and nullary operator keywords 256 on default try a variable reference or a function call 257 258parse_expression 259 call parse_non_binary to get the right hand side 260 switch on binary operator keywords to get the next operation 261 with one side if expr_none return else get the second hand 262 handle operator precedence, can call itself 263 return a map entry with the operator name as the key, and 264 left and right expression branches 265 266Part 4: Expression processing 267============================= 268 269Files: 270------ 271 - print.c (new) 272 - eval.c (new) 273 - reduce.c (new) 274 275Print: 276------ 277 278const char * 279print_expression(struct element *expr, isc_boolean_t *lose); 280const char * 281print_boolean_expression(struct element *expr, isc_boolean_t *lose); 282const char * 283print_data_expression(struct element *expr, isc_boolean_t *lose); 284const char * 285print_numeric_expression(struct element *expr, isc_boolean_t *lose); 286 287expr: expression to print 288lose: failure (??? in output) flag 289return: the text representing the expression 290 291Eval: 292----- 293 294struct element * 295eval_expression(struct element *expr, isc_boolean_t *modifiedp); 296struct element * 297eval_boolean_expression(struct element *expr, isc_boolean_t *modifiedp); 298struct element * 299eval_data_expression(struct element *expr, isc_boolean_t *modifiedp); 300struct element * 301eval_numeric_expression(struct element *expr, isc_boolean_t *modifiedp); 302 303expr: expression to evaluate 304modifiedp: a different element was returned (still false for updates 305 inside a map) 306return: the evaluated element (can have been updated for a map or a list, 307 or can be a fully different element) 308 309Evaluation is at parsing time so it is mainly a constant propagation. 310(no beta reduction for instance) 311 312Reduce: 313------- 314 315struct element * 316reduce_boolean_expression(struct element *expr); 317struct element * 318reduce_data_expression(struct element *expr); 319struct element * 320reduce_numeric_expression(struct element *expr); 321 322expr: expression to reduce 323return: NULL or the reduced expression as a Kea eval string 324 325reducing works for a limited (but interesting) set of expressions which 326can be converted to kea evaluatebool and for literals. 327 328Part 5: Specific issues 329======================= 330 331Reservations: 332------------- 333 ISC DHCP host declarations are global, Kea reservations were per subnet 334 only until 1.5. 335 It is possible to use the fixed address but: 336 - it is possible to finish with orphan reservations, i.e. 337 reservations with an address which match no subnets 338 - a reservation can have no fixed address. In this case the MA puts 339 the reservation in the last declared subnet. 340 - a reservation can have more than one fixed address and these 341 addresses can belong to different subnets. Current code pushes 342 IPv4 extra addresses in a commented extra-ip-addresses but 343 it is legal feature for IPv6. 344 - it is not easy to use prefix6 345 The use of groups in host declarations is unclear. 346 ISC DHCP UID is mapped to client-id, host-identifier to flex-id 347 Host reservation identifiers are generated on first use. 348 349Groups: 350------- 351TODO: search missing parameters from the Kea syntax. 352 (will be done in the third pass) 353 354Shared-Networks: 355---------------- 356 Waiting for the feature to be supported by Kea. 357 Currently at the end of a shared network declaration: 358 - if there is no subnets it is a fatal error 359 - if there is one subnet the shared-network is squeezed 360 - if there are more than one subnet the shared-network is commented 361TODO (useful only with Kea support for shared networks): combine permit / 362deny classes (e.g. create negation) and pop filters to subnets when 363there is one pool. 364 365Vendor-Classes and User-Classes: 366-------------------------------- 367 ISC DHCP code is inconsistent: in particular before setting the 368 super-class "tname" to "implicit-vendor-class" / "implicit-user-class" 369 it allocates a buffer for data but does not copy the lexical value 370 "val" into it... So I removed support. 371 372Classes: 373-------- 374 Only pure client-classes are supported by kea. 375 Dynamic/deleted stuff is not supported but does it make sense? 376 To spawn classes is not supported. 377 Match class selector is converted to Kea eval test when the corresponding 378 expression can be reduced. Fortunately it seems to be the common case! 379 Lease limit is not supported. 380 381Subclasses: 382----------- 383 Understood how it works: 384 - (super) class defined with a MATCH <data-expression> (vs. 385 MATCH IF <boolean-expression>) 386 - subclasses defined by <superclass-name> <data-literal> which 387 are equivalent to 388 MATCH IF <superclass-data-expression> EQUAL <data-literal> 389 So subclasses are convertible when the data expression can be reduced. 390 Cf https://kb.isc.org/article/AA-01092/202/OMAPI-support-for-classes-and-subclasses.html 391 which BTW suggests the management API could manage classes... 392 393Hardware Addresses: 394------------------- 395 Kea supports only Ethernet. 396 397Pools: 398------ 399 All permissions are not supported by Kea at the exception of class members 400 but in a very different way so not convertible. 401 Mixed DHCPv6 address and prefix pools are not supported, perhaps in this 402 case the pool should be duplicated into pool and pd-pool instances? 403 The bootp stuff was ifdef's as bootp is obsolete. 404 Temporary (aka IA_TA) is commented ny the MA. 405 ISC DHCP supports interval ranges for prefix6. Kea has a different 406 and IMHO more powerful model. 407 Pool6 permissions are not supported. 408 409Failover: 410--------- 411 Display a warning on the first use. 412 413Interfaces: 414----------- 415 Referenced interface names are pushed to an interfaces-config but it is 416 very (too!) easy to finish with a Kea config without any interface. 417 418Hostnames: 419---------- 420 ISC DHCP does dynamic resolution in parse_ip_addr_or_hostname. 421 Static (at conversion time) resolution to one address is done by 422 the MA for fixed-address. Resolution is considered as painful 423 there are better (and safer) ways to do this. The -r (resolve) 424 command line parameter controls the at-conversion-time resolution. 425 Note only the first address is returned. 426TODO: check the multiple address comment is correctly taken 427 (need a known host resolving in a stable set of addresses) 428 429Options: 430-------- 431 Some options are known only in ISC DHCP (almost fixed), a few only by Kea. 432 Formats are supposed to be the same, the only known exception 433 (DHCPv4 domain-search) was fixed by #5087. 434 For option spaces DHCPv4 vendor-encapsulated-options (code 43, in general 435 associated to vendor-class-identifier code 60) uses a dedicated feature 436 which had no equivalent in Kea (fixed). 437 Option definitions are convertible with a few exception: 438 - no support in Kea for an array of records (mainly by the lack 439 of a corresponding syntax). BTW there is no known use too. 440 - no support in Kea for an array at the end of a record (fixed) 441 All unsupported option declarations are set to full binary (X). 442 - X format means ASCII or hexa: 443 * standard options are in general mapped to binary 444 * new options are mapped to string with format x (vs x) 445 * when a string got hexadecimal data a warning in added in comments 446 suggesting to switch to plain binary. 447 - ISC DHCP use quotes for a domain-list but not for a domain-name, 448 this is no very coherent and makes domain-list different than 449 domain-name array. 450Each time an option data has a format which is not convertible than 451a CSV false binary data is produced. 452 We have no example in ISC DHCP, Kea or standard but it is possible 453 than an option defined as a fixed sized record followed by 454 (encapsulated) suboptions bugs (it already bugs toElement). 455 For operations on options ISC DHCP has supersede, send, append, 456 prepend, default (set if not yet present), Kea puts them in code order 457 with a few built-in exceptions. 458 To finish there is the way to enforce Kea to add an option in a response 459 is pretty different and can't be automatically translated (cf Kea #250). 460 461Duplicates: 462----------- 463 Many things in ISC DHCP can be duplicated: 464 - options can be redefined 465 - same host identifier used twice 466 - same fixed address used in tow different hosts 467 etc. 468 Kea is far more strict and IMHO it is a good thing. Now the MA does 469 no particular check and multiple definitions work only for classes 470 (because it is the way the ISC DHCP parse works). 471 If we have Docsis space options, they are standard in Kea so they 472 will conflict. 473 474Dynamic DNS: 475------------ 476 Details are very different so the MA maps only basic parameters 477 at the global scope. 478 479Expressions: 480------------ 481 ISC DHCP expressions are typed: boolean, numeric, and data aka string. 482 The default for a literal is to be a string so literal numbers are 483 interpreted in hexadecimal (for a strange consequence look at 484 https://kb.isc.org/article/AA-00334/56/Do-the-list-of-parameters-in-the-dhcp-parameter-request-list-need-to-be-in-hex.html ). 485 String literals are converted to string elements, hexadecimal literals 486 are converted to const-data maps. 487TODO reduce more hexa aka const-data 488 As booleans are not data there is no way to fix this: 489 /tmp/bool line 9: Expecting a data expression. 490 option ip-forwarding = foo = foo; 491 ^ 492 Cf Kea #247 493 The tautology 'foo = foo' is not a data expression so is rejected by 494 both the MA and dhcpd (BTW the role of the MA is not to fix ISC DHCP 495 shortcomings so it does what it is expected to do here). 496 Note this does not work too: 497 option ip-forwarding = true; 498 because "true" is not a keyword and it is converted into a variable 499 reference... And I expect ISC DHCP makes this true a false at runtime 500 because the variable "true" is not defined by default. 501 Reduced expressions are pretty printed to allow an extra check. 502 Hardware for DHCPv4 is expansed into a concatenation of hw-type and 503 hw-address, this allows to simplify expression where only one is used. 504 505Variables: 506---------- 507 ISC DHCP has a notion of variables in a scope where the scope can be 508 a lexical scope in the config or a scope in a function body 509 (ISC DHCP has even an unused "let" statement). 510 There is a variant of bindings for lease files using types and able 511 to recognize booleans and numbers. Unfortunately this is very specific... 512 513TODO: 514 - global host reservations 515 - class like if statement 516 - add more tests for classes in pools and class generation 517