Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove square brackets in FROM METADATA declaration #196991

Merged
merged 12 commits into from
Oct 31, 2024
Merged

Remove square brackets in FROM METADATA declaration #196991

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
995d480
remove square brackets in FROM METADATA declaration
kyracho Oct 21, 2024
235241c
fix
kyracho Oct 21, 2024
1dc5b9d
revert changes in the translation file
kyracho Oct 28, 2024
56248a9
delete repeated assertions
kyracho Oct 28, 2024
f9938bc
commit the updated esql_validation_meta_tests.json
kyracho Oct 28, 2024
d18d7c9
Merge branch 'main' into remove_square_brackets
kyracho Oct 29, 2024
92cf372
Merge branch 'main' into remove_square_brackets
stratoula Oct 29, 2024
af503fc
fix
kyracho Oct 30, 2024
568505f
fix `validation.ccs.test.ts`
kyracho Oct 30, 2024
614cb59
update `esql_validation_meta_tests.json`
kyracho Oct 30, 2024
ad3de06
fix
kyracho Oct 30, 2024
c182612
Merge branch 'main' into remove_square_brackets
stratoula Oct 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions ...esql-validation-autocomplete/src/autocomplete/__tests__/autocomplete.command.from.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,14 +91,12 @@ describe('autocomplete.suggest', () => {
test('on <kbd>SPACE</kbd> after "METADATA" keyword suggests all metadata fields', async () => {
const { assertSuggestions } = await setup();

await assertSuggestions('from a, b [METADATA /]', metadataFields);
await assertSuggestions('from a, b METADATA /', metadataFields);
kyracho marked this conversation as resolved.
Show resolved Hide resolved
});

test('on <kbd>SPACE</kbd> after "METADATA" column suggests command and pipe operators', async () => {
const { assertSuggestions } = await setup();

await assertSuggestions('from a, b [metadata _index /]', [',', '| ']);
await assertSuggestions('from a, b metadata _index /', [',', '| ']);
kyracho marked this conversation as resolved.
Show resolved Hide resolved
await assertSuggestions('from a, b metadata _index, _source /', [',', '| ']);
await assertSuggestions(`from a, b metadata ${METADATA_FIELDS.join(', ')} /`, ['| ']);
Expand All @@ -107,7 +105,6 @@ describe('autocomplete.suggest', () => {
test('filters out already used metadata fields', async () => {
const { assertSuggestions } = await setup();

await assertSuggestions('from a, b [metadata _index, /]', metadataFieldsAndIndex);
await assertSuggestions('from a, b metadata _index, /', metadataFieldsAndIndex);
kyracho marked this conversation as resolved.
Show resolved Hide resolved
});
});
Expand Down
4 changes: 2 additions & 2 deletions packages/kbn-esql-validation-autocomplete/src/autocomplete/autocomplete.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ describe('autocomplete', () => {
.map(({ name }) => name.toUpperCase() + ' $0')
);
testSuggestions(
'from a [metadata _id] | /',
'from a metadata _id | /',
commandDefinitions
.filter(({ name }) => !sourceCommands.includes(name))
.map(({ name }) => name.toUpperCase() + ' $0')
Expand All @@ -116,7 +116,7 @@ describe('autocomplete', () => {
.map(({ name }) => name.toUpperCase() + ' $0')
);
testSuggestions(
'from a [metadata _id] | eval var0 = a | /',
'from a metadata _id | eval var0 = a | /',
commandDefinitions
.filter(({ name }) => !sourceCommands.includes(name))
.map(({ name }) => name.toUpperCase() + ' $0')
Expand Down
10 changes: 0 additions & 10 deletions packages/kbn-esql-validation-autocomplete/src/definitions/options.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,16 +38,6 @@ export const metadataOption: CommandOptionsDefinition = {
skipCommonValidation: true,
validate: (option, command, references) => {
const messages: ESQLMessage[] = [];
// need to test the parent command here
if (/\[metadata/i.test(command.text)) {
messages.push(
getMessageFromId({
messageId: 'metadataBracketsDeprecation',
values: {},
locations: option.location,
})
);
}
const fields = option.args.filter(isColumnItem);
const metadataFieldsAvailable = references as unknown as Set<string>;
if (metadataFieldsAvailable.size > 0) {
Expand Down
73 changes: 14 additions & 59 deletions ...l-validation-autocomplete/src/validation/__tests__/test_suites/validation.command.from.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,73 +106,28 @@ export const validationFromCommandTestSuite = (setup: helpers.Setup) => {
await expectErrors('from index metadata _id, \t\n _index\n ', []);
});

test('errors when wrapped in brackets', async () => {
test('errors when wrapped in parentheses', async () => {
const { expectErrors } = await setup();

await expectErrors(`from index (metadata _id)`, [
"SyntaxError: mismatched input '(metadata' expecting <EOF>",
]);
});

for (const isWrapped of [true, false]) {
function setWrapping(option: string) {
return isWrapped ? `[${option}]` : option;
}

function addBracketsWarning() {
return isWrapped
? ["Square brackets '[]' need to be removed from FROM METADATA declaration"]
: [];
}

describe(`wrapped = ${isWrapped}`, () => {
test('no errors on correct usage, waning on square brackets', async () => {
const { expectErrors } = await setup();

await expectErrors(`from index ${setWrapping('METADATA _id')}`, []);
await expectErrors(
`from index ${setWrapping('METADATA _id')}`,
[],
addBracketsWarning()
);
await expectErrors(
`from index ${setWrapping('metadata _id')}`,
[],
addBracketsWarning()
);
await expectErrors(
`from index ${setWrapping('METADATA _id, _source')}`,
[],
addBracketsWarning()
);
});

test('validates fields', async () => {
const { expectErrors } = await setup();

await expectErrors(
`from index ${setWrapping('METADATA _id, _source2')}`,
[
`Metadata field [_source2] is not available. Available metadata fields are: [${METADATA_FIELDS.join(
', '
)}]`,
],
addBracketsWarning()
);
await expectErrors(
`from index ${setWrapping('metadata _id, _source')} ${setWrapping(
'METADATA _id2'
)}`,
[
isWrapped
? "SyntaxError: mismatched input '[' expecting <EOF>"
: "SyntaxError: mismatched input 'METADATA' expecting <EOF>",
],
addBracketsWarning()
);
});
describe('validates fields', () => {
test('validates fields', async () => {
const { expectErrors } = await setup();

await expectErrors(`from index METADATA _id, _source2`, [
`Metadata field [_source2] is not available. Available metadata fields are: [${METADATA_FIELDS.join(
', '
)}]`,
]);
await expectErrors(`from index metadata _id, _source METADATA _id2`, [
"SyntaxError: mismatched input 'METADATA' expecting <EOF>",
]);
});
}
});
});
});
});
Expand Down
34 changes: 7 additions & 27 deletions packages/kbn-esql-validation-autocomplete/src/validation/__tests__/validation.ccs.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,33 +24,13 @@ describe('validation', () => {
});

describe('... METADATA <indices>', () => {
for (const isWrapped of [true, false]) {
function setWrapping(option: string) {
return isWrapped ? `[${option}]` : option;
}

function addBracketsWarning() {
return isWrapped
? ["Square brackets '[]' need to be removed from FROM METADATA declaration"]
: [];
}

describe(`wrapped = ${isWrapped}`, () => {
test('no errors on correct usage, waning on square brackets', async () => {
const { expectErrors } = await setup();
await expectErrors(
`from remote-ccs:indexes ${setWrapping('METADATA _id')}`,
['Unknown index [remote-ccs:indexes]'],
addBracketsWarning()
);
await expectErrors(
`from *:indexes ${setWrapping('METADATA _id')}`,
['Unknown index [*:indexes]'],
addBracketsWarning()
);
});
});
}
test('no errors on correct usage', async () => {
const { expectErrors } = await setup();
await expectErrors(`from remote-ccs:indexes METADATA _id`, [
'Unknown index [remote-ccs:indexes]',
]);
await expectErrors(`from *:indexes METADATA _id`, ['Unknown index [*:indexes]']);
});
});
});
});
Expand Down
68 changes: 2 additions & 66 deletions packages/kbn-esql-validation-autocomplete/src/validation/esql_validation_meta_tests.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9947,70 +9947,6 @@
],
"warning": []
},
{
"query": "from index [METADATA _id]",
"error": [],
"warning": []
},
{
"query": "from index [METADATA _id]",
"error": [],
"warning": [
"Square brackets '[]' need to be removed from FROM METADATA declaration"
]
},
{
"query": "from index [metadata _id]",
"error": [],
"warning": [
"Square brackets '[]' need to be removed from FROM METADATA declaration"
]
},
{
"query": "from index [METADATA _id, _source]",
"error": [],
"warning": [
"Square brackets '[]' need to be removed from FROM METADATA declaration"
]
},
{
"query": "from index [METADATA _id, _source2]",
"error": [
"Metadata field [_source2] is not available. Available metadata fields are: [_version, _id, _index, _source, _ignored, _index_mode]"
],
"warning": [
"Square brackets '[]' need to be removed from FROM METADATA declaration"
]
},
{
"query": "from index [metadata _id, _source] [METADATA _id2]",
"error": [
"SyntaxError: mismatched input '[' expecting <EOF>"
],
"warning": [
"Square brackets '[]' need to be removed from FROM METADATA declaration"
]
},
{
"query": "from index METADATA _id",
"error": [],
"warning": []
},
{
"query": "from index METADATA _id",
"error": [],
"warning": []
},
{
"query": "from index metadata _id",
"error": [],
"warning": []
},
{
"query": "from index METADATA _id, _source",
"error": [],
"warning": []
},
{
"query": "from index METADATA _id, _source2",
"error": [
Expand All @@ -10019,11 +9955,11 @@
"warning": []
},
{
"query": "from index metadata _id, _source METADATA _id2",
"query": "from index METADATA _id, _source METADATA _id2",
"error": [
"SyntaxError: mismatched input 'METADATA' expecting <EOF>"
],
"warning": []
}
]
}
}
4 changes: 2 additions & 2 deletions packages/kbn-esql-validation-autocomplete/src/validation/validation.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1741,7 +1741,7 @@ describe('validation logic', () => {
it('should basically work when all callbacks are passed', async () => {
const allErrors = await Promise.all(
fixtures.testCases
.filter(({ query }) => query === 'from index [METADATA _id, _source2]')
.filter(({ query }) => query === 'from index METADATA _id, _source2')
.map(({ query }) =>
validateQuery(
query,
Expand All @@ -1753,7 +1753,7 @@ describe('validation logic', () => {
);
for (const [index, { errors }] of Object.entries(allErrors)) {
expect(errors.map((e) => ('severity' in e ? e.message : e.text))).toEqual(
fixtures.testCases.filter(({ query }) => query === 'from index [METADATA _id, _source2]')[
fixtures.testCases.filter(({ query }) => query === 'from index METADATA _id, _source2')[
Number(index)
].error
);
Expand Down
24 changes: 12 additions & 12 deletions packages/kbn-language-documentation/src/sections/esql_documentation_sections.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ export const initialSection = (
<Markdown
markdownContent={i18n.translate('languageDocumentation.documentationESQL.markdown', {
defaultMessage: `
An ES|QL (Elasticsearch query language) query consists of a series of commands, separated by pipe characters: \`|\`. Each query starts with a **source command**, which produces a table, typically with data from Elasticsearch.
An ES|QL (Elasticsearch query language) query consists of a series of commands, separated by pipe characters: \`|\`. Each query starts with a **source command**, which produces a table, typically with data from Elasticsearch.

A source command can be followed by one or more **processing commands**. Processing commands can change the output table of the previous command by adding, removing, and changing rows and columns.

Expand All @@ -29,7 +29,7 @@ source-command
| processing-command2
\`\`\`

The result of a query is the table produced by the final processing command.
The result of a query is the table produced by the final processing command.
`,
})}
/>
Expand Down Expand Up @@ -77,15 +77,15 @@ ES|QL can access the following metadata fields:
Use the \`METADATA\` directive to enable metadata fields:

\`\`\`
FROM index [METADATA _index, _id]
FROM index METADATA _index, _id
\`\`\`

Metadata fields are only available if the source of the data is an index. Consequently, \`FROM\` is the only source commands that supports the \`METADATA\` directive.

Once enabled, the fields are then available to subsequent processing commands, just like the other index fields:

\`\`\`
FROM ul_logs, apps [METADATA _index, _version]
FROM ul_logs, apps METADATA _index, _version
| WHERE id IN (13, 14) AND _version == 1
| EVAL key = CONCAT(_index, "_", TO_STR(id))
| SORT id, _index
Expand All @@ -95,7 +95,7 @@ FROM ul_logs, apps [METADATA _index, _version]
Also, similar to the index fields, once an aggregation is performed, a metadata field will no longer be accessible to subsequent commands, unless used as grouping field:

\`\`\`
FROM employees [METADATA _index, _id]
FROM employees METADATA _index, _id
| STATS max = MAX(emp_no) BY _index
\`\`\`
`,
Expand All @@ -114,7 +114,7 @@ FROM employees [METADATA _index, _id]
markdownContent={i18n.translate('languageDocumentation.documentationESQL.row.markdown', {
defaultMessage: `### ROW
The \`ROW\` source command produces a row with one or more columns with values that you specify. This can be useful for testing.

\`\`\`
ROW a = 1, b = "two", c = null
\`\`\`
Expand Down Expand Up @@ -207,7 +207,7 @@ ROW a = "1953-01-23T12:15:00Z - some text - 127.0.0.1"
markdownContent={i18n.translate('languageDocumentation.documentationESQL.drop.markdown', {
defaultMessage: `### DROP
Use \`DROP\` to remove columns from a table:

\`\`\`
FROM employees
| DROP height
Expand Down Expand Up @@ -347,7 +347,7 @@ ROW a = "12 15.5 15.6 true"
The \`KEEP\` command enables you to specify what columns are returned and the order in which they are returned.

To limit the columns that are returned, use a comma-separated list of column names. The columns are returned in the specified order:

\`\`\`
FROM employees
| KEEP first_name, last_name, height
Expand Down Expand Up @@ -384,7 +384,7 @@ FROM employees
{
defaultMessage: `### LIMIT
The \`LIMIT\` processing command enables you to limit the number of rows:

\`\`\`
FROM employees
| LIMIT 5
Expand All @@ -407,7 +407,7 @@ FROM employees
'languageDocumentation.documentationESQL.mvExpand.markdown',
{
defaultMessage: `### MV_EXPAND
The \`MV_EXPAND\` processing command expands multivalued fields into one row per value, duplicating other fields:
The \`MV_EXPAND\` processing command expands multivalued fields into one row per value, duplicating other fields:
\`\`\`
ROW a=[1,2,3], b="b", j=["a","b"]
| MV_EXPAND a
Expand Down Expand Up @@ -607,7 +607,7 @@ FROM employees
{
defaultMessage: `### WHERE
Use \`WHERE\` to produce a table that contains all the rows from the input table for which the provided condition evaluates to \`true\`:

\`\`\`
FROM employees
| KEEP first_name, last_name, still_hired
Expand Down Expand Up @@ -654,7 +654,7 @@ export const groupingFunctions = {
defaultMessage: `### BUCKET
Creates groups of values - buckets - out of a datetime or numeric input. The size of the buckets can either be provided directly, or chosen based on a recommended count and values range.

\`BUCKET\` works in two modes:
\`BUCKET\` works in two modes:

1. Where the size of the bucket is computed based on a buckets count recommendation (four parameters) and a range.
2. Where the bucket size is provided directly (two parameters).
Expand Down
Loading