Lines Matching full:n
16 "This tutorial is going to walk through creating a TableGen backend using Python.\n",
17 "\n",
18 … here will still apply and you could port this tutorial to any language that has a JSON parser.\n",
19 "\n",
20 "This is the process in LLVM, using a C++ backend:\n",
21 "```\n",
22 "TableGen source -> llvm-tblgen -> backend (within llvm-tblgen) -> results\n",
23 "```\n",
24 "This is what we will be doing:\n",
25 "```\n",
26 "TableGen source -> llvm-tblgen -> JSON -> Python -> results\n",
27 "```\n",
28 "\n",
29 …"The backend here is ported from one of several in \"SQLGen\" which was written by Min-Yih Hsu.\n",
30 "* SQLGen C++ sources - https://github.com/mshockwave/SQLGen\n",
31 "* LLVM dev presentation - https://www.youtube.com/watch?v=UP-LBRbvI_U\n",
32 "\n",
49 …leGen kernel. This is an iPython notebook and we're going to run `llvm-tblgen` as a subprocess.\n",
50 "\n",
61 "import os\n",
62 "import shutil\n",
63 "\n",
64 "def find_tblgen():\n",
65 " path = os.environ.get(\"LLVM_TBLGEN_EXECUTABLE\")\n",
66 " if path is not None and os.path.isfile(path) and os.access(path, os.X_OK):\n",
67 " return path\n",
68 " else:\n",
69 " path = shutil.which(\"llvm-tblgen\")\n",
70 " if path is None:\n",
71 " raise OSError(\"llvm-tblgen not found\")\n",
72 " return path\n",
73 " \n",
82 …N_EXECUTABLE` environment variable. Alternatively, edit the code to use whatever path you want.\n",
83 "\n",
97 "{\n",
98 " \"!instanceof\": {\n",
99 " \"Foo\": []\n",
100 " },\n",
101 " \"!tablegen_json_version\": 1\n",
102 "}\n"
107 "import subprocess\n",
108 "import tempfile\n",
109 "import json\n",
110 "\n",
111 "def run_tblgen(src):\n",
112 " # Passing to stdin requires a file like object.\n",
113 " with tempfile.TemporaryFile(\"w+\") as f:\n",
114 " f.write(src)\n",
115 " f.seek(0)\n",
116 " got = subprocess.run(\n",
117 " [find_tblgen(), \"--dump-json\"],\n",
118 " stdin=f,\n",
119 " stderr=subprocess.PIPE,\n",
120 " stdout=subprocess.PIPE,\n",
121 " universal_newlines=True,\n",
122 " )\n",
123 " \n",
124 " if got.stderr:\n",
125 " raise RuntimeError(\"llvm-tblgen failed with stderr: \" + got.stderr)\n",
126 " \n",
127 " return json.loads(got.stdout)\n",
128 " \n",
145 "This backend is going to generate SQL queries. The general form of a SQL query is:\n",
146 "```\n",
147 "SELECT <some field names> FROM <table name>\n",
148 " WHERE <conditions>\n",
149 " ORDER BY <field tags>;\n",
168 "query_tblgen = \"\"\"\\\n",
169 "def all;\n",
170 "def fields;\n",
171 "def none;\n",
172 "\n",
173 "def eq;\n",
174 "def ne;\n",
175 "def gt;\n",
176 "def ge;\n",
177 "def and;\n",
178 "def or;\n",
187 …ython string to fit into this notebook. We will add to this string to produce the final source.\n",
188 "\n",
189 … "This section defines some constants. First are the fields we want to get back from the query:\n",
190 "* `all` - Return all fields.\n",
191 "* `fields` - Means that we will provide a list of fields we are interested in.\n",
192 "\n",
193 …hese are string versions of various symbols. For example `ne` means `!=`, which in SQL is `<>`.\n",
194 "\n",
205 "query_tblgen += \"\"\"\\\n",
206 "class Query <string table, dag query_fields = (all), dag condition = (none)> {\n",
207 " string TableName = table;\n",
208 " dag Fields = query_fields;\n",
209 " dag WhereClause = condition;\n",
210 " list<string> OrderedBy = [];\n",
211 "}\n",
220 "Then the Query class. Its arguments are:\n",
221 "* `table` - The name of the table to query (`FROM <table>`).\n",
222 "* `query_fields` - The fields you want returned (`SELECT <fields>`).\n",
223 " * Defaults to `all` meaning return all fields.\n",
224 "* `condition` - Logic to select entries (`WHERE <conditions>`).\n",
243 "full_tblgen = query_tblgen + \"\"\"\\\n",
244 "def : Query<\"Customer\">;\n",
245 "\n",
246 "def : Query<\"Orders\", (fields \"Person\", \"Amount\")>;\n",
247 "\n",
248 "def : Query<\"Customer\", (fields \"Affiliation\"),\n",
249 " (eq \"Name\", \"Mary Blackburn\":$str)>;\n",
250 "\n",
251 "def : Query<\"Orders\", (fields \"ProductName\"),\n",
252 " (gt \"Amount\", 8)>;\n",
253 "\n",
254 "def : Query<\"Orders\", (fields \"ProductName\":$name, \"Person\"),\n",
255 " (and (gt \"Amount\", 8), (ne \"Person\", 1))> {\n",
256 " let OrderedBy = [\"$name\"];\n",
257 "}\n",
266 "Now we can define some queries. Let's go go over the last one in detail.\n",
267 "\n",
268 "```\n",
269 "def : Query<\"Orders\", (fields \"ProductName\":$name, \"Person\"),\n",
270 " (and (gt \"Amount\", 8), (ne \"Person\", 1))> {\n",
271 " let OrderedBy = [\"$name\"];\n",
272 "}\n",
273 "```\n",
274 "\n",
275 "* It will run on a table called `Orders`.\n",
276 "* We want to see the fields `ProductName` and `Person`.\n",
277 "* We have tagged `ProductName` with `$name`.\n",
278 "* The condition is that `Amount` must be greater than `8` and\n",
279 " `Person` must not be equal to `1`.\n",
280 "* The results of this query should be ordered by the field\n",
281 " tagged `$name`, which is `ProductName`.\n",
282 " \n",
283 …ic Graph) allows us to describe nested conditions. You might write this condition in Python as:\n",
284 "```\n",
285 "if (Amount > 8) and (Person != 1):\n",
286 "```\n",
287 "Putting that into a graph form:\n",
288 "```\n",
289 " |------|and|------|\n",
290 " | |\n",
291 "| Amount > 8 | | Person != 1 |\n",
292 "```\n",
314 "{\n",
315 " \"!instanceof\": {\n",
316 " \"Query\": [\n",
317 " \"anonymous_0\",\n",
318 " \"anonymous_1\",\n",
319 " \"anonymous_2\",\n",
320 " \"anonymous_3\",\n",
321 " \"anonymous_4\"\n",
322 " ]\n",
323 " },\n",
324 " \"!tablegen_json_version\": 1,\n",
325 " \"all\": {\n",
326 " \"!anonymous\": false,\n",
327 " \"!fields\": [],\n",
328 " \"!name\": \"all\",\n",
329 " \"!superclasses\": []\n",
330 " },\n",
331 " \"and\": {\n",
332 " \"!anonymous\": false,\n",
333 " \"!fields\": [],\n",
334 " \"!name\": \"and\",\n",
335 " \"!superclasses\": []\n",
336 " },\n",
337 " \"anonymous_0\": {\n",
338 " \"!anonymous\": true,\n",
339 " \"!fields\": [],\n",
340 " \"!name\": \"anonymous_0\",\n",
341 " \"!superclasses\": [\n",
342 " \"Query\"\n",
343 " ],\n",
344 " \"Fields\": {\n",
345 " \"args\": [],\n",
346 " \"kind\": \"dag\",\n",
347 " \"operator\": {\n",
348 " \"def\": \"all\",\n",
349 " \"kind\": \"def\",\n",
350 " \"printable\": \"all\"\n",
351 " },\n",
352 " \"printable\": \"(all)\"\n",
353 " },\n",
354 " \"OrderedBy\": [],\n",
355 " \"TableName\": \"Customer\",\n",
356 " \"WhereClause\": {\n",
357 " \"args\": [],\n",
358 " \"kind\": \"dag\",\n",
359 " \"operator\": {\n",
360 " \"def\": \"none\",\n",
361 " \"kind\": \"def\",\n",
362 " \"printable\": \"none\"\n",
363 " },\n",
364 " \"printable\": \"(none)\"\n",
365 " }\n",
366 " },\n",
367 " \"anonymous_1\": {\n",
368 " \"!anonymous\": true,\n",
369 " \"!fields\": [],\n",
370 " \"!name\": \"anonymous_1\",\n",
371 " \"!superclasses\": [\n",
372 " \"Query\"\n",
373 " ],\n",
374 " \"Fields\": {\n",
375 " \"args\": [\n",
376 " [\n",
377 " \"Person\",\n",
378 " null\n",
379 " ],\n",
380 " [\n",
381 " \"Amount\",\n",
382 " null\n",
383 " ]\n",
384 " ],\n",
385 " \"kind\": \"dag\",\n",
386 " \"operator\": {\n",
387 " \"def\": \"fields\",\n",
388 " \"kind\": \"def\",\n",
389 " \"printable\": \"fields\"\n",
390 " },\n",
391 " \"printable\": \"(fields \\\"Person\\\", \\\"Amount\\\")\"\n",
392 " },\n",
393 " \"OrderedBy\": [],\n",
394 " \"TableName\": \"Orders\",\n",
395 " \"WhereClause\": {\n",
396 " \"args\": [],\n",
397 " \"kind\": \"dag\",\n",
398 " \"operator\": {\n",
399 " \"def\": \"none\",\n",
400 " \"kind\": \"def\",\n",
401 " \"printable\": \"none\"\n",
402 " },\n",
403 " \"printable\": \"(none)\"\n",
404 " }\n",
405 " },\n",
406 " \"anonymous_2\": {\n",
407 " \"!anonymous\": true,\n",
408 " \"!fields\": [],\n",
409 " \"!name\": \"anonymous_2\",\n",
410 " \"!superclasses\": [\n",
411 " \"Query\"\n",
412 " ],\n",
413 " \"Fields\": {\n",
414 " \"args\": [\n",
415 " [\n",
416 " \"Affiliation\",\n",
417 " null\n",
418 " ]\n",
419 " ],\n",
420 " \"kind\": \"dag\",\n",
421 " \"operator\": {\n",
422 " \"def\": \"fields\",\n",
423 " \"kind\": \"def\",\n",
424 " \"printable\": \"fields\"\n",
425 " },\n",
426 " \"printable\": \"(fields \\\"Affiliation\\\")\"\n",
427 " },\n",
428 " \"OrderedBy\": [],\n",
429 " \"TableName\": \"Customer\",\n",
430 " \"WhereClause\": {\n",
431 " \"args\": [\n",
432 " [\n",
433 " \"Name\",\n",
434 " null\n",
435 " ],\n",
436 " [\n",
437 " \"Mary Blackburn\",\n",
438 " \"str\"\n",
439 " ]\n",
440 " ],\n",
441 " \"kind\": \"dag\",\n",
442 " \"operator\": {\n",
443 " \"def\": \"eq\",\n",
444 " \"kind\": \"def\",\n",
445 " \"printable\": \"eq\"\n",
446 " },\n",
447 " \"printable\": \"(eq \\\"Name\\\", \\\"Mary Blackburn\\\":$str)\"\n",
448 " }\n",
449 " },\n",
450 " \"anonymous_3\": {\n",
451 " \"!anonymous\": true,\n",
452 " \"!fields\": [],\n",
453 " \"!name\": \"anonymous_3\",\n",
454 " \"!superclasses\": [\n",
455 " \"Query\"\n",
456 " ],\n",
457 " \"Fields\": {\n",
458 " \"args\": [\n",
459 " [\n",
460 " \"ProductName\",\n",
461 " null\n",
462 " ]\n",
463 " ],\n",
464 " \"kind\": \"dag\",\n",
465 " \"operator\": {\n",
466 " \"def\": \"fields\",\n",
467 " \"kind\": \"def\",\n",
468 " \"printable\": \"fields\"\n",
469 " },\n",
470 " \"printable\": \"(fields \\\"ProductName\\\")\"\n",
471 " },\n",
472 " \"OrderedBy\": [],\n",
473 " \"TableName\": \"Orders\",\n",
474 " \"WhereClause\": {\n",
475 " \"args\": [\n",
476 " [\n",
477 " \"Amount\",\n",
478 " null\n",
479 " ],\n",
480 " [\n",
481 " 8,\n",
482 " null\n",
483 " ]\n",
484 " ],\n",
485 " \"kind\": \"dag\",\n",
486 " \"operator\": {\n",
487 " \"def\": \"gt\",\n",
488 " \"kind\": \"def\",\n",
489 " \"printable\": \"gt\"\n",
490 " },\n",
491 " \"printable\": \"(gt \\\"Amount\\\", 8)\"\n",
492 " }\n",
493 " },\n",
494 " \"anonymous_4\": {\n",
495 " \"!anonymous\": true,\n",
496 " \"!fields\": [],\n",
497 " \"!name\": \"anonymous_4\",\n",
498 " \"!superclasses\": [\n",
499 " \"Query\"\n",
500 " ],\n",
501 " \"Fields\": {\n",
502 " \"args\": [\n",
503 " [\n",
504 " \"ProductName\",\n",
505 " \"name\"\n",
506 " ],\n",
507 " [\n",
508 " \"Person\",\n",
509 " null\n",
510 " ]\n",
511 " ],\n",
512 " \"kind\": \"dag\",\n",
513 " \"operator\": {\n",
514 " \"def\": \"fields\",\n",
515 " \"kind\": \"def\",\n",
516 " \"printable\": \"fields\"\n",
517 " },\n",
518 " \"printable\": \"(fields \\\"ProductName\\\":$name, \\\"Person\\\")\"\n",
519 " },\n",
520 " \"OrderedBy\": [\n",
521 " \"$name\"\n",
522 " ],\n",
523 " \"TableName\": \"Orders\",\n",
524 " \"WhereClause\": {\n",
525 " \"args\": [\n",
526 " [\n",
527 " {\n",
528 " \"args\": [\n",
529 " [\n",
530 " \"Amount\",\n",
531 " null\n",
532 " ],\n",
533 " [\n",
534 " 8,\n",
535 " null\n",
536 " ]\n",
537 " ],\n",
538 " \"kind\": \"dag\",\n",
539 " \"operator\": {\n",
540 " \"def\": \"gt\",\n",
541 " \"kind\": \"def\",\n",
542 " \"printable\": \"gt\"\n",
543 " },\n",
544 " \"printable\": \"(gt \\\"Amount\\\", 8)\"\n",
545 " },\n",
546 " null\n",
547 " ],\n",
548 " [\n",
549 " {\n",
550 " \"args\": [\n",
551 " [\n",
552 " \"Person\",\n",
553 " null\n",
554 " ],\n",
555 " [\n",
556 " 1,\n",
557 " null\n",
558 " ]\n",
559 " ],\n",
560 " \"kind\": \"dag\",\n",
561 " \"operator\": {\n",
562 " \"def\": \"ne\",\n",
563 " \"kind\": \"def\",\n",
564 " \"printable\": \"ne\"\n",
565 " },\n",
566 " \"printable\": \"(ne \\\"Person\\\", 1)\"\n",
567 " },\n",
568 " null\n",
569 " ]\n",
570 " ],\n",
571 " \"kind\": \"dag\",\n",
572 " \"operator\": {\n",
573 " \"def\": \"and\",\n",
574 " \"kind\": \"def\",\n",
575 " \"printable\": \"and\"\n",
576 " },\n",
577 " \"printable\": \"(and (gt \\\"Amount\\\", 8), (ne \\\"Person\\\", 1))\"\n",
578 " }\n",
579 " },\n",
580 " \"eq\": {\n",
581 " \"!anonymous\": false,\n",
582 " \"!fields\": [],\n",
583 " \"!name\": \"eq\",\n",
584 " \"!superclasses\": []\n",
585 " },\n",
586 " \"fields\": {\n",
587 " \"!anonymous\": false,\n",
588 " \"!fields\": [],\n",
589 " \"!name\": \"fields\",\n",
590 " \"!superclasses\": []\n",
591 " },\n",
592 " \"ge\": {\n",
593 " \"!anonymous\": false,\n",
594 " \"!fields\": [],\n",
595 " \"!name\": \"ge\",\n",
596 " \"!superclasses\": []\n",
597 " },\n",
598 " \"gt\": {\n",
599 " \"!anonymous\": false,\n",
600 " \"!fields\": [],\n",
601 " \"!name\": \"gt\",\n",
602 " \"!superclasses\": []\n",
603 " },\n",
604 " \"ne\": {\n",
605 " \"!anonymous\": false,\n",
606 " \"!fields\": [],\n",
607 " \"!name\": \"ne\",\n",
608 " \"!superclasses\": []\n",
609 " },\n",
610 " \"none\": {\n",
611 " \"!anonymous\": false,\n",
612 " \"!fields\": [],\n",
613 " \"!name\": \"none\",\n",
614 " \"!superclasses\": []\n",
615 " },\n",
616 " \"or\": {\n",
617 " \"!anonymous\": false,\n",
618 " \"!fields\": [],\n",
619 " \"!name\": \"or\",\n",
620 " \"!superclasses\": []\n",
621 " }\n",
622 "}\n"
627 "full_json = run_tblgen(full_tblgen)\n",
649 "{'Query': ['anonymous_0', 'anonymous_1', 'anonymous_2', 'anonymous_3', 'anonymous_4']}\n"
675 "['Query']\n"
701 …nd': 'dag', 'operator': {'def': 'all', 'kind': 'def', 'printable': 'all'}, 'printable': '(all)'}\n"
730 …is looping over all defs of a certain class and outputting some text based on their properties.\n",
731 "\n",
745 "['anonymous_0', 'anonymous_1', 'anonymous_2', 'anonymous_3', 'anonymous_4']\n"
750 "def find_all_queries(j):\n",
751 " queries = []\n",
752 " for key in j:\n",
753 " # ! means it is some metadata, not a def.\n",
754 " if not key.startswith(\"!\"):\n",
755 " value = full_json[key]\n",
756 " # If we inherit from Query.\n",
757 " if \"Query\" in value[\"!superclasses\"]:\n",
758 " queries.append(value)\n",
759 " return queries\n",
760 "\n",
761 "queries = find_all_queries(full_json)\n",
762 " \n",
771 …d and `llvm-tblgen` just came up with a name for us. For this purpose the names are irrelevant.\n",
772 "\n",
786 " AND \n"
791 "def emit_operator(operator):\n",
792 " return {\n",
793 " 'gt': ' > ',\n",
794 " 'ge': ' >= ',\n",
795 " 'lt': ' < ',\n",
796 " 'le': ' <= ',\n",
797 " 'ne': ' <> ',\n",
798 " 'eq': ' = ',\n",
799 " 'or': ' OR ',\n",
800 " 'and': ' AND '\n",
801 " }[operator]\n",
802 "\n",
824 "Abc, Def\n"
829 "def emit_fields(args):\n",
830 " # Return a comma separated list of arg names.\n",
831 " return \", \".join([arg[0] for arg in args])\n",
832 "\n",
854 "Name = \"Mary Blackburn\"\n"
859 "from collections.abc import Mapping\n",
860 "\n",
861 "def emit_where_clause(where_clause):\n",
862 " output = \"\"\n",
863 " num_args = len(where_clause[\"args\"])\n",
864 " \n",
865 " for idx, arg in enumerate(where_clause[\"args\"]):\n",
866 " arg_name, arg_type = arg\n",
867 "\n",
868 " if isinstance(arg_name, Mapping):\n",
869 " # This is a nested where clause.\n",
870 " output += emit_where_clause(arg_name)\n",
871 " else:\n",
872 " # This is some condition.\n",
873 " if arg_type == \"str\":\n",
874 " # String types must be emitted with \"\" around them.\n",
875 " output += '\"' + arg_name + '\"'\n",
876 " else:\n",
877 " output += str(arg_name)\n",
878 "\n",
879 " # If this is not the last arg, emit the condition.\n",
880 " if idx != (num_args-1):\n",
881 " output += emit_operator(where_clause[\"operator\"][\"def\"])\n",
882 " \n",
883 " return output\n",
884 "\n",
885 "print(emit_where_clause({\n",
886 "\"args\": [[\"Name\",None], \n",
887 " [\"Mary Blackburn\", \"str\"]],\n",
888 "\"kind\": \"dag\",\n",
889 "\"operator\": {\n",
890 " \"def\": \"eq\",\n",
891 " \"kind\": \"def\",\n",
892 " \"printable\": \"eq\"\n",
901 …we will find a possible mix of conditions and other DAGS. We recurse to handle the latter case.\n",
902 "\n",
916 "\n",
917 " ORDER BY ABC, DEF\n"
922 "def emit_ordered_by(ordered_by, field_tag_map):\n",
923 " # No ORDER BY statement to emit.\n",
924 " if not ordered_by:\n",
925 " return \"\"\n",
926 " \n",
927 " output = \"\\n ORDER BY \"\n",
928 " num_ordered_by = len(ordered_by)\n",
929 " \n",
930 " for idx, field_name in enumerate(ordered_by):\n",
931 " # If it is a tag\n",
932 " if field_name.startswith('$'):\n",
933 " # Find the corresponding field name\n",
934 " tag_name = field_name[1:]\n",
935 " field_name = field_tag_map.get(tag_name)\n",
936 " if field_name is None:\n",
937 " raise RuntimeError('Unrecognized tag \"{}\"'.format(\n",
938 " tag_name))\n",
939 "\n",
940 " # Separate each tag after the first with \", \".\n",
941 " if idx != 0:\n",
942 " output += \", \"\n",
943 " output += field_name\n",
944 " \n",
945 " return output\n",
946 "\n",
955 …ng return nothing, otherwise loop over all the fields we want to order by and emit their names.\n",
956 "\n",
970 "{'abc': 'ABC', 'def': 'DEF'}\n"
975 "def build_tag_map(arguments):\n",
976 " # Args are [Name, Tag]. Reverse this so we have [Tag, Name].\n",
977 " # Add each one to a dictionary where Tag is the key and Name is the value.\n",
978 " return dict([reversed(a) for a in arguments])\n",
979 "\n",
990 "def emit_query(q):\n",
991 " fields_init = q[\"Fields\"]\n",
992 " field_op_name = fields_init[\"operator\"][\"def\"]\n",
993 " if not field_op_name in [\"all\", \"fields\"]:\n",
994 " raise RuntimeError(\"Invalid dag operator \" + field_op_name)\n",
995 " \n",
996 " field_tag_map = build_tag_map(fields_init[\"args\"])\n",
997 " \n",
998 " where_clause = q[\"WhereClause\"]\n",
999 " has_where = where_clause[\"operator\"][\"def\"] != \"none\"\n",
1000 " \n",
1001 " ret = \"SELECT \"\n",
1002 " if field_op_name == \"all\":\n",
1003 " ret += \"*\"\n",
1004 " ret += emit_fields(fields_init[\"args\"])\n",
1005 " ret += \" FROM \" + q[\"TableName\"]\n",
1006 " if has_where:\n",
1007 " ret += \"\\n WHERE \" + emit_where_clause(where_clause)\n",
1008 " ret += emit_ordered_by(q[\"OrderedBy\"], field_tag_map)\n",
1009 " ret += \";\"\n",
1010 " \n",
1040 "SELECT * FROM Customer;\n",
1041 "\n",
1042 "SELECT Person, Amount FROM Orders;\n",
1043 "\n",
1044 "SELECT Affiliation FROM Customer\n",
1045 " WHERE Name = \"Mary Blackburn\";\n",
1046 "\n",
1047 "SELECT ProductName FROM Orders\n",
1048 " WHERE Amount > 8;\n",
1049 "\n",
1050 "SELECT ProductName, Person FROM Orders\n",
1051 " WHERE Amount > 8 AND Person <> 1\n",
1052 " ORDER BY ProductName;\n",
1053 "\n"
1058 "for q in queries:\n",
1059 " print(emit_query(q) + \"\\n\")"
1067 …ow we run `emit_query` and print out the results. There you have it, that's a TableGen backend!\n",
1068 "\n",
1069 …this case it was SQL queries. In LLVM it's most often C++ code but it can be anything you want.\n",
1070 "\n",