Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(web-analytics): Add backend version of web analytics queries #17629

Merged
merged 13 commits into from
Oct 2, 2023
Merged

feat(web-analytics): Add backend version of web analytics queries #17629

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
9249be7
Add web top sources query to the backend
robbie-c Sep 26, 2023
663e802
Get top source query working
robbie-c Sep 26, 2023
51f52e7
Add other queries and share code between query runners
robbie-c Sep 27, 2023
e2bfc8a
Fix ABC warnings
robbie-c Sep 27, 2023
c8701d0
Tidy up folder structure of query runners
robbie-c Sep 28, 2023
d1236f2
Rename verbose class name
robbie-c Sep 28, 2023
336da52
Revert method order
robbie-c Sep 28, 2023
7c1054f
Rename web analytics classes to have web in the name
robbie-c Sep 29, 2023
dbf41d8
Update posthog/hogql_queries/query_runner.py
robbie-c Sep 29, 2023
9698c04
Improve query runner type names
robbie-c Sep 29, 2023
27c2483
Fix broken import
robbie-c Sep 29, 2023
669dd33
Fix response types
robbie-c Oct 2, 2023
c6ba505
Add comment explaining why we set the hash key
robbie-c Oct 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions frontend/src/queries/nodes/DataTable/DataTable.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -511,9 +511,9 @@ export function DataTable({ uniqueKey, query, setQuery, context, cachedResults }
}
footer={
canLoadNextData &&
((response as any).results.length > 0 || !responseLoading) && (
<LoadNext query={query.source} />
)
((response as any).results.length > 0 ||
(response as any).result.length > 0 ||
!responseLoading) && <LoadNext query={query.source} />
}
/>
)}
Expand Down
10 changes: 8 additions & 2 deletions frontend/src/queries/nodes/DataTable/dataTableLogic.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,9 +138,15 @@ export const dataTableLogic = kea<dataTableLogicType>([
}))
}

return response && 'results' in response && Array.isArray(response.results)
? response.results.map((result: any) => ({ result })) ?? null
const results = !response
? null
: 'results' in response && Array.isArray(response.results)
? response.results
: 'result' in response && Array.isArray(response.result)
? response.result
: null
Comment on lines +141 to 147
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to find a way to standardise here 🙈

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, hogql and hogql insights were already different here 🙉 I could just change one but I'm a bit scared of sweeping changes like that. Probably safest to change the hogql insights given it's not running in prod yet, and could do that in a different PR to keep this one less noisy.


return results ? results.map((result: any) => ({ result })) ?? null : null
},
],
queryWithDefaults: [
Expand Down
14 changes: 13 additions & 1 deletion frontend/src/queries/nodes/DataTable/queryFeatures.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,11 @@
import { isEventsQuery, isHogQLQuery, isPersonsNode } from '~/queries/utils'
import {
isEventsQuery,
isHogQLQuery,
isPersonsNode,
isWebTopClicksQuery,
isWebTopPagesQuery,
isWebTopSourcesQuery,
} from '~/queries/utils'
import { Node } from '~/queries/schema'

export enum QueryFeature {
Expand Down Expand Up @@ -40,5 +47,10 @@ export function getQueryFeatures(query: Node): Set<QueryFeature> {
features.add(QueryFeature.personsSearch)
}

if (isWebTopSourcesQuery(query) || isWebTopPagesQuery(query) || isWebTopClicksQuery(query)) {
features.add(QueryFeature.columnsInResponse)
features.add(QueryFeature.resultIsArrayOfArrays)
}

return features
}
181 changes: 181 additions & 0 deletions frontend/src/queries/schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,15 @@
},
{
"$ref": "#/definitions/TimeToSeeDataSessionsQuery"
},
{
"$ref": "#/definitions/WebTopSourcesQuery"
},
{
"$ref": "#/definitions/WebTopClicksQuery"
},
{
"$ref": "#/definitions/WebTopPagesQuery"
}
]
},
Expand Down Expand Up @@ -389,6 +398,15 @@
},
{
"$ref": "#/definitions/TimeToSeeDataSessionsQuery"
},
{
"$ref": "#/definitions/WebTopSourcesQuery"
},
{
"$ref": "#/definitions/WebTopClicksQuery"
},
{
"$ref": "#/definitions/WebTopPagesQuery"
}
],
"description": "Source of the events"
Expand Down Expand Up @@ -2269,6 +2287,169 @@
},
"required": ["result"],
"type": "object"
},
"WebAnalyticsFilters": {},
"WebTopClicksQuery": {
"additionalProperties": false,
"properties": {
"dateRange": {
"$ref": "#/definitions/DateRange"
},
"filters": {
"$ref": "#/definitions/WebAnalyticsFilters"
},
"kind": {
"const": "WebTopClicksQuery",
"type": "string"
},
"response": {
"$ref": "#/definitions/WebTopClicksQueryResponse"
}
},
"required": ["kind", "filters"],
"type": "object"
},
"WebTopClicksQueryResponse": {
"additionalProperties": false,
"properties": {
"columns": {
"items": {},
"type": "array"
},
"is_cached": {
"type": "boolean"
},
"last_refresh": {
"type": "string"
},
"next_allowed_client_refresh": {
"type": "string"
},
"result": {
"items": {},
"type": "array"
},
"timings": {
"items": {
"$ref": "#/definitions/QueryTiming"
},
"type": "array"
},
"types": {
"items": {},
"type": "array"
}
},
"required": ["result"],
"type": "object"
},
"WebTopPagesQuery": {
"additionalProperties": false,
"properties": {
"dateRange": {
"$ref": "#/definitions/DateRange"
},
"filters": {
"$ref": "#/definitions/WebAnalyticsFilters"
},
"kind": {
"const": "WebTopPagesQuery",
"type": "string"
},
"response": {
"$ref": "#/definitions/WebTopPagesQueryResponse"
}
},
"required": ["kind", "filters"],
"type": "object"
},
"WebTopPagesQueryResponse": {
"additionalProperties": false,
"properties": {
"columns": {
"items": {},
"type": "array"
},
"is_cached": {
"type": "boolean"
},
"last_refresh": {
"type": "string"
},
"next_allowed_client_refresh": {
"type": "string"
},
"result": {
"items": {},
"type": "array"
},
"timings": {
"items": {
"$ref": "#/definitions/QueryTiming"
},
"type": "array"
},
"types": {
"items": {},
"type": "array"
}
},
"required": ["result"],
"type": "object"
},
"WebTopSourcesQuery": {
"additionalProperties": false,
"properties": {
"dateRange": {
"$ref": "#/definitions/DateRange"
},
"filters": {
"$ref": "#/definitions/WebAnalyticsFilters"
},
"kind": {
"const": "WebTopSourcesQuery",
"type": "string"
},
"response": {
"$ref": "#/definitions/WebTopSourcesQueryResponse"
}
},
"required": ["kind", "filters"],
"type": "object"
},
"WebTopSourcesQueryResponse": {
"additionalProperties": false,
"properties": {
"columns": {
"items": {},
"type": "array"
},
"is_cached": {
"type": "boolean"
},
"last_refresh": {
"type": "string"
},
"next_allowed_client_refresh": {
"type": "string"
},
"result": {
"items": {},
"type": "array"
},
"timings": {
"items": {
"$ref": "#/definitions/QueryTiming"
},
"type": "array"
},
"types": {
"items": {},
"type": "array"
}
},
"required": ["result"],
"type": "object"
}
}
}
57 changes: 56 additions & 1 deletion frontend/src/queries/schema.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,11 @@ export enum NodeKind {
StickinessQuery = 'StickinessQuery',
LifecycleQuery = 'LifecycleQuery',

// Web analytics queries
WebTopSourcesQuery = 'WebTopSourcesQuery',
WebTopPagesQuery = 'WebTopPagesQuery',
WebTopClicksQuery = 'WebTopClicksQuery',

// Time to see data
TimeToSeeDataSessionsQuery = 'TimeToSeeDataSessionsQuery',
TimeToSeeDataQuery = 'TimeToSeeDataQuery',
Expand All @@ -75,6 +80,9 @@ export type AnyDataNode =
| HogQLQuery
| HogQLMetadata
| TimeToSeeDataSessionsQuery
| WebTopSourcesQuery
| WebTopClicksQuery
| WebTopPagesQuery

export type QuerySchema =
// Data nodes (see utils.ts)
Expand Down Expand Up @@ -277,7 +285,15 @@ export type HasPropertiesNode = EventsNode | EventsQuery | PersonsNode
export interface DataTableNode extends Node, DataTableNodeViewProps {
kind: NodeKind.DataTableNode
/** Source of the events */
source: EventsNode | EventsQuery | PersonsNode | HogQLQuery | TimeToSeeDataSessionsQuery
source:
| EventsNode
| EventsQuery
| PersonsNode
| HogQLQuery
| TimeToSeeDataSessionsQuery
| WebTopSourcesQuery
| WebTopClicksQuery
| WebTopPagesQuery

/** Columns shown in the table, unless the `source` provides them. */
columns?: HogQLExpression[]
Expand Down Expand Up @@ -483,6 +499,45 @@ export interface LifecycleQuery extends InsightsQueryBase {
response?: LifecycleQueryResponse
}

export type WebAnalyticsFilters = any
mariusandra marked this conversation as resolved.
Show resolved Hide resolved

export interface WebAnalyticsQueryBase {
dateRange?: DateRange
}

export interface WebTopSourcesQuery extends WebAnalyticsQueryBase {
kind: NodeKind.WebTopSourcesQuery
filters: WebAnalyticsFilters
response?: WebTopSourcesQueryResponse
}
export interface WebTopSourcesQueryResponse extends QueryResponse {
result: unknown[]
types?: unknown[]
columns?: unknown[]
}

export interface WebTopClicksQuery extends WebAnalyticsQueryBase {
kind: NodeKind.WebTopClicksQuery
filters: WebAnalyticsFilters
response?: WebTopClicksQueryResponse
}
export interface WebTopClicksQueryResponse extends QueryResponse {
result: unknown[]
types?: unknown[]
columns?: unknown[]
}

export interface WebTopPagesQuery extends WebAnalyticsQueryBase {
kind: NodeKind.WebTopPagesQuery
filters: WebAnalyticsFilters
response?: WebTopPagesQueryResponse
}
export interface WebTopPagesQueryResponse extends QueryResponse {
result: unknown[]
types?: unknown[]
columns?: unknown[]
}

export type InsightQueryNode =
| TrendsQuery
| FunnelsQuery
Expand Down
15 changes: 15 additions & 0 deletions frontend/src/queries/utils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@ import {
TimeToSeeDataJSONNode,
DatabaseSchemaQuery,
SavedInsightNode,
WebTopSourcesQuery,
WebTopClicksQuery,
WebTopPagesQuery,
} from '~/queries/schema'
import { TaxonomicFilterGroupType, TaxonomicFilterValue } from 'lib/components/TaxonomicFilter/types'
import { dayjs } from 'lib/dayjs'
Expand Down Expand Up @@ -90,6 +93,18 @@ export function isHogQLQuery(node?: Node | null): node is HogQLQuery {
return node?.kind === NodeKind.HogQLQuery
}

export function isWebTopSourcesQuery(node?: Node | null): node is WebTopSourcesQuery {
return node?.kind === NodeKind.WebTopSourcesQuery
}

export function isWebTopClicksQuery(node?: Node | null): node is WebTopClicksQuery {
return node?.kind === NodeKind.WebTopClicksQuery
}

export function isWebTopPagesQuery(node?: Node | null): node is WebTopPagesQuery {
return node?.kind === NodeKind.WebTopPagesQuery
}

export function containsHogQLQuery(node?: Node | null): boolean {
if (!node) {
return false
Expand Down
2 changes: 1 addition & 1 deletion frontend/src/scenes/organization/Settings/VerifiedDomains/SSOSelect.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import { SSO_PROVIDER_NAMES } from 'lib/constants'
import { preflightLogic } from 'scenes/PreflightCheck/preflightLogic'
import { SSOProvider } from '~/types'

interface SSOSelectInterface {
export interface SSOSelectInterface {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Drive-by fix, fixing a TypeScript error in some test code (which therefore doesn't trigger CI failures, but does still show up in my editor)

value: SSOProvider | ''
loading: boolean
onChange: (value: SSOProvider | '') => void
Expand Down
Loading
Loading