Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[source-slack] IncrementalMessageStream for threads channel never stops syncing #430

Open
1 task
whimo opened this issue Oct 22, 2024 · 0 comments
Open
1 task

[source-slack] IncrementalMessageStream for threads channel never stops syncing #430

whimo opened this issue Oct 22, 2024 · 0 comments

Comments

@whimo
Copy link

whimo commented Oct 22, 2024

Connector Name

source-slack

Connector Version

1.3.1

What step the error happened?

During the sync

Relevant information

Threads channel syncing never stops. It seems to cycle through all the channels, updating the end timestamp each loop (as some time passed while the data was queried), and does that over and over.
Other channels work fine.

Using PyAirbyte:

import airbyte as ab

source = ab.get_source(
    "source-slack",
    config={
        "api_token": "<my-api-token>",
        "start_date": "2024-01-01T00:00:00Z",
        "lookback_window": 1,
        "join_channels": False,
    },
    install_if_missing=True,
)
source.check()
source.select_streams(["channels", "channel_members", "channel_messages", "threads"])


result = source.read()

for name, records in result.streams.items():
    print(f"Stream {name}: {len(list(records))} records")
    for record in records:
        print(record.get("text"))

Relevant log output

2024-10-22 15:40:57 - INFO - Check succeeded
2024-10-22 15:41:02 - INFO - Starting syncing SourceSlack
2024-10-22 15:41:02 - INFO - Marking stream channels as STARTED
2024-10-22 15:41:02 - INFO - Setting state of SourceSlack stream to {'__ab_no_cursor_state_message': True}
2024-10-22 15:41:02 - INFO - Syncing stream: channels 
2024-10-22 15:41:02 - INFO - Marking stream channels as RUNNING
2024-10-22 15:41:02 - INFO - Read 4 records from channels stream
2024-10-22 15:41:02 - INFO - Marking stream channels as STOPPED
2024-10-22 15:41:02 - INFO - Finished syncing channels
2024-10-22 15:41:02 - INFO - SourceSlack runtimes:
Syncing stream channels 0:00:00.365530
2024-10-22 15:41:02 - INFO - Marking stream channel_members as STARTED
2024-10-22 15:41:02 - INFO - Setting state of SourceSlack stream to {'__ab_no_cursor_state_message': True}
2024-10-22 15:41:02 - INFO - Syncing stream: channel_members 
2024-10-22 15:41:02 - INFO - Marking stream channel_members as RUNNING
2024-10-22 15:41:03 - INFO - Read 18 records from channel_members stream
2024-10-22 15:41:03 - INFO - Marking stream channel_members as STOPPED
2024-10-22 15:41:03 - INFO - Finished syncing channel_members
2024-10-22 15:41:03 - INFO - SourceSlack runtimes:
Syncing stream channel_members 0:00:00.985201
Syncing stream channels 0:00:00.365530
2024-10-22 15:41:03 - INFO - Marking stream channel_messages as STARTED
2024-10-22 15:41:03 - INFO - Setting state of SourceSlack stream to {'states': [{'partition': {'channel': 'C07Q9KX2U6T', 'parent_slice': {}}, 'cursor': {'float_ts': 1729324525.094199}}, {'partition': {'channel': 'C07QHGM9B1C', 'parent_slice': {}}, 'cursor': {'float_ts': 1729324525.118059}}, {'partition': {'channel': 'C07RCUS2MGQ', 'parent_slice': {}}, 'cursor': {'float_ts': 1729324525.155319}}, {'partition': {'channel': 'C07RFRXUQN5', 'parent_slice': {}}, 'cursor': {'float_ts': 1729283939.911469}}]}
2024-10-22 15:41:03 - INFO - Syncing stream: channel_messages 
2024-10-22 15:41:03 - INFO - Marking stream channel_messages as RUNNING
2024-10-22 15:41:04 - INFO - Read 6 records from channel_messages stream
2024-10-22 15:41:04 - INFO - Marking stream channel_messages as STOPPED
2024-10-22 15:41:04 - INFO - Finished syncing channel_messages
2024-10-22 15:41:04 - INFO - SourceSlack runtimes:
Syncing stream channel_members 0:00:00.985201
Syncing stream channel_messages 0:00:01.123485
Syncing stream channels 0:00:00.365530
2024-10-22 15:41:04 - INFO - Marking stream threads as STARTED
2024-10-22 15:41:04 - INFO - Setting state of SourceSlack stream to {}
2024-10-22 15:41:04 - INFO - Syncing stream: threads 
2024-10-22 15:41:04 - INFO - Reading the channel: `test-slack-ingestion`
2024-10-22 15:41:04 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:04 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:05 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1721347200.0, 'latest': 1729600864.626089}
2024-10-22 15:41:05 - INFO - Marking stream threads as RUNNING
2024-10-22 15:41:07 - INFO - Reading the channel: `social`
2024-10-22 15:41:07 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:07 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:07 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1721347200.0, 'latest': 1729600867.384261}
2024-10-22 15:41:08 - INFO - Reading the channel: `all-new-workspace`
2024-10-22 15:41:08 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:09 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:09 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1721347200.0, 'latest': 1729600868.85753}
2024-10-22 15:41:10 - INFO - Reading the channel: `support`
2024-10-22 15:41:10 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:10 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:10 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1721347200.0, 'latest': 1729600870.328075}
2024-10-22 15:41:12 - INFO - Reading the channel: `test-slack-ingestion`
2024-10-22 15:41:12 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:12 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:12 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1721347200.0, 'latest': 1729600872.45988}
2024-10-22 15:41:14 - INFO - Reading the channel: `social`
2024-10-22 15:41:14 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:14 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:14 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1721347200.0, 'latest': 1729600874.548036}
2024-10-22 15:41:15 - INFO - Reading the channel: `all-new-workspace`
2024-10-22 15:41:15 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:15 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:15 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1721347200.0, 'latest': 1729600875.567569}
2024-10-22 15:41:16 - INFO - Reading the channel: `support`
2024-10-22 15:41:16 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:16 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:16 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1721347200.0, 'latest': 1729600876.609494}
2024-10-22 15:41:18 - INFO - Reading the channel: `test-slack-ingestion`
2024-10-22 15:41:18 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:18 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:18 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1721347200.0, 'latest': 1729600878.290911}
2024-10-22 15:41:20 - INFO - Reading the channel: `social`
2024-10-22 15:41:20 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:20 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:20 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1721347200.0, 'latest': 1729600880.449153}
2024-10-22 15:41:21 - INFO - Reading the channel: `all-new-workspace`
2024-10-22 15:41:21 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:21 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:21 - INFO - Syncing replies {'channel': 'C07RCUS2MGQ', 'oldest': 1721347200.0, 'latest': 1729600881.485752}
2024-10-22 15:41:22 - INFO - Reading the channel: `support`
2024-10-22 15:41:22 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:22 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:22 - INFO - Syncing replies {'channel': 'C07RFRXUQN5', 'oldest': 1721347200.0, 'latest': 1729600882.548746}
2024-10-22 15:41:24 - INFO - Reading the channel: `test-slack-ingestion`
2024-10-22 15:41:24 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:24 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:24 - INFO - Syncing replies {'channel': 'C07Q9KX2U6T', 'oldest': 1721347200.0, 'latest': 1729600884.416193}
2024-10-22 15:41:27 - INFO - Reading the channel: `social`
2024-10-22 15:41:27 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1704067200.0, 'latest': 1712707200.0}
2024-10-22 15:41:27 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1712707200.0, 'latest': 1721347200.0}
2024-10-22 15:41:27 - INFO - Syncing replies {'channel': 'C07QHGM9B1C', 'oldest': 1721347200.0, 'latest': 1729600887.079985}

Contribute

  • Yes, I want to contribute
@marcosmarxm marcosmarxm transferred this issue from airbytehq/airbyte Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@whimo