[email protected]: support for builtin resources, support for "*" in ads q…

…ueries

* bq-writer: Fixed: writer failed with 'dataset not found' error if dataset exists in a different location than specified
* console-writer: Fixed: writer went to the infinite recursion if the only one column didn't fit into console width after transposing

Change-Id: I4b832ec04aa26d7759921c8f3b643f2df1f6e445

README.md

@@ -309,9 +309,7 @@ Please see the [README](gcp/README.md) there for all information.
 ## Differences in Python and NodeJS versions
 
 ### Query syntax and features
-There are differences in which features supported for queries.
-Python-only features:
-* pre-process query files as Jinja templates
+There are differences in supported features for queries.
 
 NodeJS-only features:
 * expressions (${...})
@@ -325,12 +323,11 @@ NodeJS on the contrary parses queries and initializes BigQuery schema before exe
 There are differences in BigQuery table structures as well.
 Python version creates one table per script. While NodeJS creates a table per script per customer and then creates a view to combine all customer tables.
 For example, you have a query campaign.sql. As a result you'll get a querable source 'campaign' in BigQuery in any way. But for Python version it'll be a table.
-For NodeJS it'll be a view like `create view dataset.campaign as select * from campaign_* when _TABLE_PREFIX in (cid1,cid2)`, where cid1, cid2 are customer id you supplied.
+For NodeJS it'll be a view like `create view dataset.campaign as select * from campaign_* when _TABLE_PREFIX in (cid1,cid2)`, where cid1, cid2 are customer ids you supplied.
 
 From Ads API we can get arrays, structs and arrays of arrays or structs. In Python version all arrays will be degrated to string with "|" separator.
-In NodeJS version the result will be a repeated field (array).
+In NodeJS version the result by default will be a repeated field (array) but can be degrated to string with separator via the `bq.array-handling` option.
 If values of an array from Ads API are also arrays or structs, they will be converted to JSON.
-Though in NodeJS you can tune how to process arrays. The described behavior is by default. You can change it via `bq.array-handling` argument.
 
 ### API support
 Python version supports any API version (currently available).

docs/how-to-write-queries.md

@@ -12,9 +12,10 @@
    - [Macros in virtual columns](#macros-in-virtual-columns)
    - [Common macros](#common-macros)
  - [Templates](#templates)
- - [SQL parameters](#sql-parameters)
  - [Expressions and Macros](#expressions-and-macros)
+ - [Functions](#functions)
  - [Built-in queries](#built-in-queries)
+ - [SQL parameters](#sql-parameters)
 
 
 ## Intro
@@ -53,7 +54,6 @@ FROM resource
 * Aliases (`AS column_name`)
 * Nested resources (`:nested.resource.name`)
 * Resource indices (`~position`)
-* Functions (`:$func`) - only in Node.js
 * Virtual columns (`metric.name / metric.name_2 AS alias`)
 
 ### Aliases
@@ -171,7 +171,7 @@ When this query is executed it's expected that two macros `--macros.start_date=.
 
 ### Macros in virtual columns
 
-Macros can be excluded not only in WHERE statements as in the example above but also in the SELECT statement.
+Macros can be used not only in WHERE statements as in the example above but also in the SELECT statement.
 In that case this macros will be expanded and then treated as a virtual column.
 
 ```
@@ -226,26 +226,9 @@ FROM asset_performance
 
 When this query is executed it's expected to have template `--template.cohort_days=0,1,3,7` is supplied to `gaarf`.
 
-Please note that all values passed through CLI arguments are strings. But there's a special case - a value containing ","
+Please note that all values passed through CLI arguments are strings. But there's a special case - a value containing "," - it's converted to an array.
 It will create 4 columns (named `installs_0_day`, `installs_1_day`, etc).
 
-## SQL Parameters
-
-You can use normal sql type parameters with `sql` argument (NodeJS only):
-```
-SELECT *
-FROM {dst_dataset}.{table-src}
-WHERE name LIKE @name
-```
-and to execute:
-`gaarf-bq --macro.table-src=table1 --macro.dst_dataset=dataset1 --sql.name='myname%'`
-
-it will create a parameterized query to run in BQ:
-```
-SELECT *
-FROM dataset1.table1
-WHERE name LIKE @name
-```
 
 ## Expressions and Macros
 > *Note*: currently expressions are supported only in NodeJS version.
@@ -336,6 +319,31 @@ ${date(2022,7,20).plusMonths(1)}
 output: "2022-08-20"
 
 
+## Functions
+
+NodeJS version support in-place functions in JavaScript.
+The functions support consists of two parts:
+* function execution - it's suffix `:$func` at any select field (where func is a funciton name)
+* function definition - a block below the main query starting with `FUNCTIONS` after which a normal JS code with a function definition follows
+
+Example:
+```
+SELECT
+  campaign.id AS campaign_id,
+  campaign_criterion.ad_schedule.day_of_week:$format AS ad_schedule_day_of_week
+FROM campaign_criterion
+FUNCTIONS
+function format(val) {
+  let days = ['пн', 'вт', 'ср', 'чт', 'пт', 'сб', 'вс'];
+  if (!val) return '';
+  return val === 8 ? days[6] : days[val-2];
+}
+```
+
+Here we defined a function `format` (converts a numeric week day into a localized day name) and then called it for `campaign_criterion.ad_schedule.day_of_week` column.
+So functions always accept only one parameter - a field value.
+
+
 ## Built-in queries
 
 Google Ads API Report Fetcher can also works with built-in queries, which use the following syntax:
@@ -348,3 +356,22 @@ It expacts to provide a built-in query name when selecting from special `builtin
 
 Currently the following queries are  available:
 * `ocid_mapping` - return  `account_id` and `ocid` from each child account under MCC; `ocid` can be used to build links to Google Ads UI.
+
+
+## SQL Parameters
+
+You can use normal sql type parameters with `sql` argument (NodeJS only):
+```
+SELECT *
+FROM {dst_dataset}.{table-src}
+WHERE name LIKE @name
+```
+and to execute:
+`gaarf-bq --macro.table-src=table1 --macro.dst_dataset=dataset1 --sql.name='myname%'`
+
+it will create a parameterized query to run in BQ:
+```
+SELECT *
+FROM dataset1.table1
+WHERE name LIKE @name
+```

js/CHANGELOG.md

@@ -1,4 +1,10 @@
 # Changelog
+## 2.5 - 2023-10-04
+* support for builtin resources
+* support for the `*` selector in Ads queries - it expands to all scalar primitive and enum fields of a resource
+* bq-writer: Fixed: writer failed with 'dataset not found' error if dataset exists in a different location than specified
+* console-writer: Fixed: writer went to the infinite recursion if the only one column didn't fit into console width after transposing
+
 ## 2.4 - 2023-09-22
 * support Google Ads API v14.1 (updated google-ads-api to v14.1)