RefreshIndicator doesn't work with panelBuilder

[JoakimMellonn]

Describe the bug
When using a RefreshIndicator inside the panelBuilder, you can't pull down to use the RefreshIndicator. I have tested that the code I'm using works by itself as expected.

To Reproduce
Steps to reproduce the behavior:

• Place a RefreshIndicator inside the panelBuilder.
• Use a ListView Builder to make it scrollable.

Expected behavior
When the panel is all the way down and you pull down, the RefreshIndicator should do its thing.

Smartphone (please complete the following information):

• Device: iOS Simulator
• OS: iOS 15.2
• Flutter Version: 3.0.5
• sliding_up_panel Version: 2.0.0

Sample main.dart
Please provide a sample main.dart that reproduces this issue. The code provided here should be able to be run on its own without any external dependencies.

import 'package:flutter/material.dart';
import 'package:sliding_up_panel/sliding_up_panel.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({Key? key}) : super(key: key);

  // This widget is the root of your application.
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: const WithSlidingPanel(), //It works without using SlidingPanel, but not with.
    );
  }
}

class WithSlidingPanel extends StatelessWidget {
  const WithSlidingPanel({Key? key}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      backgroundColor: Theme.of(context).colorScheme.background,
      body: SlidingUpPanel(
        minHeight: MediaQuery.of(context).size.height * 0.3,
        maxHeight: MediaQuery.of(context).size.height * 0.9,
        panelBuilder: recordingList,
        body: body(context),
      ),
    );
  }

  Widget body(BuildContext context) {
    return const Center(
      child: Text("I'm the body!"),
    );
  }

  Widget recordingList(ScrollController scrollController) {
    return RefreshIndicator(
      onRefresh: () async {},
      child: ListView.builder(
        controller: scrollController,
        itemCount: 10,
        itemBuilder: (context, index) {
          return Container(
            color: Colors.blue,
            height: 100,
            child: Center(
              child: Text("I'm widget number $index"),
            ),
          );
        },
      ),
    );
  }
}

class WithoutSlidingPanel extends StatelessWidget {
  const WithoutSlidingPanel({Key? key}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      body: RefreshIndicator(
        onRefresh: () async {},
        child: ListView.builder(
          itemCount: 10,
          itemBuilder: (context, index) {
            return Container(
              color: Colors.blue,
              height: 100,
              child: Center(
                child: Text("I'm widget number $index"),
              ),
            );
          },
        ),
      ),
    );
  }
}