Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"IndexError: no such group" in python driver when using with superset and partitionned tables #45149

Open
renaudk opened this issue May 6, 2024 · 6 comments
Open

"IndexError: no such group" in python driver when using with superset and partitionned tables #45149

renaudk opened this issue May 6, 2024 · 6 comments
Labels
type/bug Something isn't working

Comments

@renaudk
Copy link

renaudk commented May 6, 2024 •
edited
Loading

Steps to reproduce the behavior (Required)

  1. Create a partitionned table with primary key
CREATE TABLE IF NOT EXISTS mydata (
	id BIGINT,
	created_on DATETIME
)
ENGINE=OLAP
PRIMARY KEY(`id`, `created_on`)
PARTITION BY date_trunc('year', `created_on`)
DISTRIBUTED BY HASH(`id`, `created_on`);
  1. Try to add dataset in superset

Expected behavior (Required)

Should work

Real behavior (Required)

Superset show error: An Error Occurred - Unable to load columns for the selected table. Please select a different table.
superset-error

Output:

2024-05-06 16:39:43,982:ERROR:flask_appbuilder.api:no such group
Traceback (most recent call last):
  File "/root/venv/lib/python3.10/site-packages/flask_appbuilder/api/__init__.py", line 110, in wraps
    return f(self, *args, **kwargs)
  File "/root/venv/lib/python3.10/site-packages/superset/views/base_api.py", line 127, in wraps
    raise ex
  File "/root/venv/lib/python3.10/site-packages/superset/views/base_api.py", line 121, in wraps
    duration, response = time_function(f, self, *args, **kwargs)
  File "/root/venv/lib/python3.10/site-packages/superset/utils/core.py", line 1463, in time_function
    response = func(*args, **kwargs)
  File "/root/venv/lib/python3.10/site-packages/superset/utils/log.py", line 255, in wrapper
    value = f(*args, **kwargs)
  File "/root/venv/lib/python3.10/site-packages/superset/databases/api.py", line 742, in table_metadata
    table_info = get_table_metadata(database, table_name, schema_name)
  File "/root/venv/lib/python3.10/site-packages/superset/databases/utils.py", line 67, in get_table_metadata
    columns = database.get_columns(table_name, schema_name)
  File "/root/venv/lib/python3.10/site-packages/superset/models/core.py", line 839, in get_columns
    return self.db_engine_spec.get_columns(
  File "/root/venv/lib/python3.10/site-packages/superset/db_engine_specs/base.py", line 1340, in get_columns
    cast(list[SQLAColumnType], inspector.get_columns(table_name, schema))
  File "/root/venv/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py", line 497, in get_columns
    col_defs = self.dialect.get_columns(
  File "<string>", line 2, in get_columns
  File "/root/venv/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py", line 55, in cache
    ret = fn(self, con, *args, **kw)
  File "/root/venv/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/base.py", line 2835, in get_columns
    parsed_state = self._parsed_state_or_create(
  File "/root/venv/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/base.py", line 3085, in _parsed_state_or_create
    return self._setup_parser(
  File "<string>", line 2, in _setup_parser
  File "/root/venv/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py", line 55, in cache
    ret = fn(self, con, *args, **kw)
  File "/root/venv/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/base.py", line 3121, in _setup_parser
    return parser.parse(sql, charset)
  File "/root/venv/lib/python3.10/site-packages/starrocks/reflection.py", line 58, in parse
    self._parse_table_options(re.split(r"\r?\n", show_create.rsplit(') ')[-1]), state)
  File "/root/venv/lib/python3.10/site-packages/starrocks/reflection.py", line 223, in _parse_table_options
    skip_lines = self._parse_partition_desc(lines, index, state)
  File "/root/venv/lib/python3.10/site-packages/starrocks/reflection.py", line 301, in _parse_partition_desc
    state.table_options["%s_%s" % (self.dialect.name, "partition_by")] = m.group("partition")
IndexError: no such group

StarRocks version (Required)

  • 3.2.6-2585333

Apache Superset version

  • 4.0.0

StarRocks python driver

  • 1.0.6
@renaudk renaudk added the type/bug Something isn't working label May 6, 2024
@renaudk
Copy link
Author

renaudk commented May 6, 2024

As a workaround, I tried to comment line 300 & 301 of starrocks/reflection.py and the error has gone.
Is there any risk of side effect?

@mustafa-qamaruddin
Copy link

Same issue on partitioned tables in superset 4.0.2 and Starrocks Version: 3.3.0

$ pip freeze | grep "starrocks"
starrocks==1.0.6

@jaogoy
Copy link
Contributor

jaogoy commented Jul 17, 2024

As a workaround, I tried to comment line 300 & 301 of starrocks/reflection.py and the error has gone. Is there any risk of side effect?

Do you know the cause, and can you fix it?
What't the error message in FE or BE?

@mustafa-qamaruddin
Copy link

It only happens with the partitioned tables defined as follow:

CREATE TABLE IF NOT EXISTS transactions
(
    Id           varchar(256),
    TnsYear      int not null,
    TnsMonth     int not null,
    TnsDayOfWeek int,
    TnsHr        int,
    Channel      varchar(256),
    StoreId      varchar(256),
    CustomerId   varchar(256),
    TnsDate      date not null
)
PARTITION BY date_trunc('month', tnsDate)
DISTRIBUTED BY HASH (tnsDate)
order by (StoreId, CustomerId);


2024-07-24 18:24:18 2024-07-24 16:24:18,659:ERROR:flask_appbuilder.api:no such group
2024-07-24 18:24:18 Traceback (most recent call last):
2024-07-24 18:24:18   File "/usr/local/lib/python3.10/site-packages/flask_appbuilder/api/__init__.py", line 110, in wraps
2024-07-24 18:24:18     return f(self, *args, **kwargs)
2024-07-24 18:24:18   File "/app/superset/views/base_api.py", line 127, in wraps
2024-07-24 18:24:18     raise ex
2024-07-24 18:24:18   File "/app/superset/views/base_api.py", line 121, in wraps
2024-07-24 18:24:18     duration, response = time_function(f, self, *args, **kwargs)
2024-07-24 18:24:18   File "/app/superset/utils/core.py", line 1463, in time_function
2024-07-24 18:24:18     response = func(*args, **kwargs)
2024-07-24 18:24:18   File "/app/superset/utils/log.py", line 255, in wrapper
2024-07-24 18:24:18     value = f(*args, **kwargs)
2024-07-24 18:24:18   File "/app/superset/databases/api.py", line 741, in table_metadata
2024-07-24 18:24:18     table_info = get_table_metadata(database, table_name, schema_name)
2024-07-24 18:24:18   File "/app/superset/databases/utils.py", line 67, in get_table_metadata
2024-07-24 18:24:18     columns = database.get_columns(table_name, schema_name)
2024-07-24 18:24:18   File "/app/superset/models/core.py", line 847, in get_columns
2024-07-24 18:24:18     return self.db_engine_spec.get_columns(
2024-07-24 18:24:18   File "/app/superset/db_engine_specs/base.py", line 1343, in get_columns
2024-07-24 18:24:18     cast(list[SQLAColumnType], inspector.get_columns(table_name, schema))
2024-07-24 18:24:18   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py", line 497, in get_columns
2024-07-24 18:24:18     col_defs = self.dialect.get_columns(
2024-07-24 18:24:18   File "<string>", line 2, in get_columns
2024-07-24 18:24:18   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py", line 55, in cache
2024-07-24 18:24:18     ret = fn(self, con, *args, **kw)
2024-07-24 18:24:18   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/base.py", line 2832, in get_columns
2024-07-24 18:24:18     parsed_state = self._parsed_state_or_create(
2024-07-24 18:24:18   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/base.py", line 3074, in _parsed_state_or_create
2024-07-24 18:24:18     return self._setup_parser(
2024-07-24 18:24:18   File "<string>", line 2, in _setup_parser
2024-07-24 18:24:18   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py", line 55, in cache
2024-07-24 18:24:18     ret = fn(self, con, *args, **kw)
2024-07-24 18:24:18   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/mysql/base.py", line 3110, in _setup_parser
2024-07-24 18:24:18     return parser.parse(sql, charset)
2024-07-24 18:24:18   File "/app/superset_home/.local/lib/python3.10/site-packages/starrocks/reflection.py", line 58, in parse
2024-07-24 18:24:18     self._parse_table_options(re.split(r"\r?\n", show_create.rsplit(') ')[-1]), state)
2024-07-24 18:24:18   File "/app/superset_home/.local/lib/python3.10/site-packages/starrocks/reflection.py", line 223, in _parse_table_options
2024-07-24 18:24:18     skip_lines = self._parse_partition_desc(lines, index, state)
2024-07-24 18:24:18   File "/app/superset_home/.local/lib/python3.10/site-packages/starrocks/reflection.py", line 301, in _parse_partition_desc
2024-07-24 18:24:18     state.table_options["%s_%s" % (self.dialect.name, "partition_by")] = m.group("partition")
2024-07-24 18:24:18 IndexError: no such group

@mustafa-qamaruddin
Copy link

And here is the only warning I see on the fe.logs, but not sure if it is relevant to the error above:

2024-07-24 18:15:53.116+02:00 WARN (ForkJoinPool.commonPool-worker-7|1169) [StmtExecutor.executeStmtWithExecPlan():2405] Failed to execute executeStmtWithExecPlan
com.starrocks.common.UserException: global dict greater than DICT_DECODE_MAX_SIZE
	at com.starrocks.qe.DefaultCoordinator.getNext(DefaultCoordinator.java:818) ~[starrocks-fe.jar:?]
	at com.starrocks.qe.StmtExecutor.executeStmtWithExecPlan(StmtExecutor.java:2399) ~[starrocks-fe.jar:?]
	at com.starrocks.statistic.StatisticExecutor.queryDictSync(StatisticExecutor.java:213) ~[starrocks-fe.jar:?]
	at com.starrocks.sql.optimizer.statistics.CacheDictManager$1.lambda$asyncLoad$0(CacheDictManager.java:74) ~[starrocks-fe.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) ~[?:?]
	at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692) ~[?:?]
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) ~[?:?]
	at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) ~[?:?]
	at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) ~[?:?]
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) ~[?:?]
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) ~[?:?]

@maver1ck
Copy link

maver1ck commented Oct 16, 2024 •
edited
Loading

I just added to Superset Dockerfile
RUN sed -i '300,301d' /usr/local/lib/python3.10/site-packages/starrocks/reflection.py

Problem is not related to BE and FE. Just wrong parsing of CREATE TABLE statement by driver.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mustafa-qamaruddin @maver1ck @jaogoy @renaudk