Pass arguments to a sensor

[petarzoric]

Hi Guys,

I am building a sensor that connects to a GitLab repository and triggers whenever there is a new commit.
I would like to make the sensor configurable, so that I can pass a git branch to the sensor.

@sensor(name="git_sensor", job=some_job)
def git_commit_sensor(context):
    headers = {"PRIVATE-TOKEN": GITLAB_TOKEN}
    response = requests.get(GITLAB_API_BASE, headers=headers)
    if response.status_code == 200:
        commits = response.json()
        if not commits:
            return SkipReason("No new commits found")
        
        latest_commit = commits[0]
        commit_id = latest_commit['id']
        commit_message = latest_commit['message']

        # Get the previous commit ID from the cursor
        last_commit_id = context.cursor

        if last_commit_id!= commit_id:
            print(f"fetched new commit with message {commit_message}")
            context.update_cursor(commit_id)
            return RunRequest(run_key=commit_id)
    else:
        context.log.error(f"Failed to fetch commits: {response.status_code} - {response.text}")`

I would like to pass in the git branch in a way like this:

@sensor(name="git_sensor", job=some_job)
def git_commit_sensor(context, git_branch):`

How can I achieve this?

[garethbrickman]

Check out using resources in sensors: https://docs.dagster.io/concepts/partitions-schedules-sensors/sensors#using-resources-in-sensors

[petarzoric]

Thanks for pointing me towards the right direction. I managed to pass the branch to my sensor via resources. 👍
However, I am not sure if I am using the resources concept correctly; at least the way I used it feels rather unintuitive to me:

My sensor definition now looks like this:

@sensor(name="git_sensor", job=train_mnist_model_job)
def git_commit_sensor(context, mainBranch: GitBranchResource):

My resources file:

class GitBranchResource(ConfigurableResource):
    branch_name: str

My definitions:

defs = Definitions(
    assets=all_assets,
    jobs=[train_mnist_model_job],
    sensors=[sensors.git_commit_changed_dvc_sensor, sensors.git_commit_sensor],
    resources={
        "mainBranch": GitBranchResource(branch_name="main"),
        "devBranch": GitBranchResource(branch_name="develop"),
        "testSensorBranch": GitBranchResource(branch_name="feature/testSensor"),
        },
)

Is there a more elegant way to customize the branch via dagster resources? Having to create a dedicated resource for each branch feels a bit annoying, but maybe I am not using resources correctly? Would appreciate some feedback.

Thanks!

[garethbrickman]

That looks correct to me. Taking a step back, is there a reason why the branch name needs to be "passed in" this way? Couldn't they be listed in the sensor code itself? Or should you compartmentalize and have one sensor per branch?