ela
Contents

ManualPackages

Tables from JSON, YAML, TOML and CSV

In the terminalela --spec tables-from-json-yaml-toml-and-csv

Each reader goes straight from text to a table: json.table, yaml.table, toml.table and csv.table. It is the path an API's answer or an export takes most often, in one call and at a fraction of the memory: json.table reads the text once and keeps only the rows' cells, never the document as maps and lists.

body = $(curl -sS https://shop.example/api/orders)t = json.table($body, at = "data.orders",    columns = {"id": "id", "name": "customer.name", "city": "customer.city", "created": "created"},    types = {"created": Type.TIME}, fallback = {"city": "unknown"})

json.table(text, at = path, columns = {name: path}, flatten = false, types = …, fallback = {column: value}), and yaml.table and toml.table the same:

  • at = is the path to the list of rows, in the path syntax of Nested data: "data.orders", or the list form for a key with a dot. Left out, the document itself is the list. A path that leads nowhere is an error naming the step that failed, as tree.get's is, and one that leads to a map or a plain value an error that says so; a document that is a map is an error suggesting its first list, at = "data.orders". A row is a map, and any other row is an error naming it.
  • Without columns =, every key of row 0 is a column, in its order, and every row has the same keys: one more or one fewer is an error naming the row and the key, unless fallback = names that column's value. A key that holds a map or a list is an error naming the key and showing columns = with its parts, columns = {"customer.name": "customer.name", "customer.city": "customer.city"}.
  • flatten = true makes a nested map's keys columns of their own, named by their paths: customer.name, customer.address.city. At most 256 columns: more is an error giving the count and pointing to columns =. A list inside a row stays an error, pointing to tree.pick, or a table of its own with at =. flatten and columns are not given together.
  • columns = {name: path} makes a column of each path, in the order written, as tree.pick does: a row the path doesn't reach is an error naming the row, the path and the step, unless fallback = names that column's value, which stands in for null too. A path that reaches a map or a list is an error showing the paths into it.
  • A column takes the type of its values, null allowed in any; a column whose values differ in type is an error naming the row, the column and both types, pointing to types =. types = takes table.of's form, {"id": Type.NUMBER, "created": Type.TIME}, and converts each value: the text "12" to a number, ISO 8601 text to a time, a number or a bool to its text for Type.STRING. A value that doesn't convert is an error at its row.
  • A key in fallback that is no column, an at = or a column's path written wrong, and flatten with columns are mistakes, which try never catches; every other error is the data's, which try catches.
  • A fault in the text is said first, at its line and column, as decode says it, wherever it is.
t = json.table($body, at = "data.orders")# error: json.table: row 0's key "customer" holds a map, and a cell is text, a number, a bool or null: name its parts as columns, columns = {"customer.name": "customer.name", "customer.city": "customer.city"}, or give flatten = truet = json.table($body, at = "data.orders", flatten = true)$t.columns   # ["id", "created", "customer.name", "customer.city"]

csv.table(text, header = true, sep = ",", columns = null, types = null, layouts = null) is table.of(csv.decode(text, sep = sep), …) in one call, with table.of's arguments: the first row names the columns unless header = false, columns = [...] names or renames them, every field is text until types = gives its column a type, and layouts = a time column's layout. A CSV file is already rows, so it takes no at =.

t = csv.table(file.read("orders.csv"), sep = ";", types = {"amount": Type.NUMBER})
  • json.table builds no tree and no row: what lies off the path to the rows is read and passed over, and each value in a row goes straight into its column. 200,000 orders, 45 MB of JSON, make a table of five columns in 0.17 s at 61 MB at the peak, the text itself 45 MB, where json.decode, tree.pick and table.of take 0.68 s and 675 MB. A number too large for ela off the path is passed over with the rest.
  • yaml.table reads the text the same way, through amber-yaml's events, an alias and a merge key as yaml.decode builds them; toml.table takes the rows of an array of tables, [[orders]], the same way, through amber-toml's. 160,000 orders, 20 MB of YAML, make a table of six columns in 0.54 s at 44 MB, where it took 0.68 s and 303 MB; 20 MB of TOML in 0.17 s at 44 MB, where it took 0.48 s and 498 MB. Rows TOML gives in another shape (an inline array of tables, a row with a dotted key or a table of its own) are read from the decoded document, as before.