[flutterfire_ui] : multi-query support to deal with Firestore 10 limit when using IN operator

[azazadev]

it's possible to support List<Query> in FirestoreListView to WA Firestore 10 limit when using IN operator ?

Feature description

Take this example

collections :

posts 
	- id
	- message
	- date
	- userId
	- ...
	- ...

users :
	- userId
	- name
	- ...
	- ...

scenario : user have 100 friends

Expected : FirestoreListView with all posts of 100 friends

below code will not work due to Firestore 10 limit when using IN operator

final postsQuery = FirebaseFirestore.instance.collection('posts').Where("userId", "in", friends (list of 100 ids ))

FirestoreListView<Map<String, dynamic>>(
  query: postsQuery,
  itemBuilder: (context, snapshot) {
   ...
   ...
  },
);

Possible WA :

// Split 100 users in list of 10 
final postsQuery1 = FirebaseFirestore.instance.collection('posts').Where("userId", "in", friends (list of 10 ids ))
final postsQuery2 = FirebaseFirestore.instance.collection('posts').Where("userId", "in", friends (list of 10 ids ))
final postsQuery3 = FirebaseFirestore.instance.collection('posts').Where("userId", "in", friends (list of 10 ids ))
...
final postsQuery10 = FirebaseFirestore.instance.collection('posts').Where("userId", "in", friends (list of 10 ids ))

FirestoreListView<Map<String, dynamic>>(
  queries: [postsQuery1,postsQuery2, ... postsQuery10],
  itemBuilder: (context, snapshot) {
   ...
   ...
  },
);

[ahmednfwela]

I usually workaround this by using rxdart and StreamBuilder

[azazadev]

I usually workaround this by using rxdart and StreamBuilder

Thanks @ahmednfwela !, can you please provide example ? this will be helpful

[ahmednfwela]

here is a simple widget that solves your specific case:

// @dart=2.14
import 'package:cloud_firestore/cloud_firestore.dart';
import 'package:flutter/material.dart';
import 'package:rxdart/rxdart.dart';
import 'package:quiver/iterables.dart' as quiver;

//typedef for readability
typedef DocumentType = Map<String, dynamic>;
typedef SnapshotType = QueryDocumentSnapshot<DocumentType>;

class MultiFirestore extends StatelessWidget {
  const MultiFirestore({Key? key}) : super(key: key);

  Stream<List<SnapshotType>> getPostsQuery(
    Iterable<String> userIds,
  ) {
    final postsCollecton = FirebaseFirestore.instance.collection('posts');
    //1: get unique users
    final uniqueUsers = userIds.toSet();
    //2: generate chunked queries based on uniqueUsers
    final splitQueries = quiver
        .partition(uniqueUsers, 10)
        .map((e) => postsCollecton.where('userId', whereIn: e));
    //3: combine them with rxdart
    return Rx.combineLatest<QuerySnapshot<DocumentType>, List<SnapshotType>>(
      splitQueries.map((e) => e.snapshots()),
      (values) => quiver.merge(values.map((e) => e.docs)).toList(),
    );
  }

  List<String> mockFriendsList() {
    return List.generate(100, (index) => 'friendUserId');
  }

  @override
  Widget build(BuildContext context) {
    return StreamBuilder<List<SnapshotType>>(
      stream: getPostsQuery(mockFriendsList()),
      builder: (context, snapshot) {
        final data = snapshot.data;
        if (data == null) return CircularProgressIndicator();
        return ListView.builder(
          itemCount: data.length,
          itemBuilder: (context, index) {
            final item = data[index];
            //whatever widget here
            return SizedBox.shrink();
          },
        );
      },
    );
  }
}

Disclaimer: this DOES NOT support server-side pagination, since you are grabbing data from different queries in parallel

[azazadev]

In my case I'm using pagination ( paginate_firestore 1.0.3 package ), and I want to move to the new FlutterFire UI and this feature is missed on both ( paginate_firestore and FlutterFire UI )

hope your example helps someone, Thanks again

[ahmednfwela]

well, if you absolutely want pagination that bad, you can use nested ListViews

// @dart=2.14
import 'package:cloud_firestore/cloud_firestore.dart';
import 'package:flutter/material.dart';
import 'package:flutterfire_ui/firestore.dart';

class UserModel {
  final String uid;
  final String name;

  UserModel({
    required this.uid,
    required this.name,
  });
}

class PostModel {
  final String postId;
  final String uid;
  final String message;
  final DateTime date;
  PostModel({
    required this.postId,
    required this.uid,
    required this.message,
    required this.date,
  });
}

class MultiFirestore extends StatelessWidget {
  const MultiFirestore({Key? key}) : super(key: key);

  Query<PostModel> getPostsQuery(String userId) {
    //use your own conversion here
    final postsCollecton = FirebaseFirestore.instance
        .collection('posts')
        .withConverter(
          fromFirestore: (snapshot, options) =>
              PostModel(uid: '', message: '', postId: '', date: DateTime.now()),
          toFirestore: (value, options) => {},
        );
    //use your own query here, order by date to get predicatable pagination
    return postsCollecton
        .where('uid', isEqualTo: userId)
        .orderBy('date', descending: true);
  }

  List<UserModel> mockFriendsList() {
    return List.generate(
      100,
      (index) => UserModel(
        name: 'Friend Name Id$index',
        uid: index.toString(),
      ),
    );
  }

  @override
  Widget build(BuildContext context) {
    final friends = mockFriendsList();
    //you can also paginate the friends query if you want !
    return ListView.builder(
      itemCount: friends.length,
      itemBuilder: (context, index) {
        final item = friends[index];
        return FirestoreListView<PostModel>(
          //these two are important
          shrinkWrap: true,
          physics: NeverScrollableScrollPhysics(),
          //========================
          query: getPostsQuery(item.uid),
          pageSize: 10, //supports pagination as well !
          itemBuilder: (context, doc) {
            //build your post here
            return Placeholder();
          },
        );
      },
    );
  }
}

this is the most optimal solution I can think of for your situation

[hyochan]

I hope the solution I made in gist helps newcomers.

[max-anders]

Hello, Thanks for the suggestions, but I can't figure it out, I implemented a pagination system and I'm now trying to filter the pagination by a follow list (user ids) so that the user only sees the posts of the users he follows,

Here's the relevant part, as the user can follow more than 10 users, it seems that the whereIn clause isn't suitable.
Can anyone suggest how to adapt this method?

I'd appreciate any guidance

The problem is the same with FirestoreListView

Filter using this
final followingDocs = await _users.doc(uid).collection('following').get();
final followingUids = followingDocs.docs.map((doc) => doc.id).toList();

Future<(List<Post>, DocumentSnapshot?)> fetchUserPosts(PostParams params,
      {pageSize, lastDoc}) async {
    // Fetch all tracks
    var query = _posts
        .where('isPrivate', isEqualTo: false)
        .orderBy('createdAt', descending: true)
        .limit(pageSize);

    query = lastDoc != null ? query.startAfterDocument(lastDoc) : query;

    if (params.type != null) {
      query = query.where('type', isEqualTo: params.type);
    }

    if (params.uid != null) {
      query = query.where('uid', isEqualTo: params.uid);
    }

    final snapshots = await query.get();
    var docs = snapshots.docs;

    List<Post> posts = [];
    DocumentSnapshot? lastDocument;

    for (var doc in docs) {
      posts.add(Post.fromMap(doc.data() as Map<String, dynamic>));
    }

    if (docs.isNotEmpty) {
      lastDocument = docs.last;
    }

    return (posts, lastDocument);
  }