How to scrape more than 120 Search Result's for a place

[Chetan11-dev]

By default, the Google Maps API limits search results to 120 per query and restricts the ability to scroll beyond this number. To obtain more than 120 search results, one must employ a strategy involving the zoom levels and geographic coordinates, and using multiple queries. This approach is done in three primary steps:

1. Determine Search Points: The first step involves identifying a list of geographic coordinates and corresponding zoom levels for the desired search area. For instance, to cover various parts of Bangalore, one might use the following sets of coordinates and zoom levels:

   [
       {
           "place": "bangalore",
           "geo_coordinates": "12.900490, 77.571466",
           "zoom": 14
       },
       {
           "place": "bangalore",
           "geo_coordinates": "12.906849, 77.577990",
           "zoom": 14
       }
   ]

2. Execute Scraper: The next step is to run the scraper for each combination of coordinates and zoom levels to collect the search results.

3. Consolidate Results: Finally, the collected results need to be merged.

The second and third steps are relatively straightforward. However, the first step, involving the determination of search points, presents a challenge. If one can devise a solution to this problem, it becomes possible to scrape more than 120 search results. This step will likely require using an API, such as the Nominatim API, to accurately determine the coordinates of a place.

For those interested in solving this issue, a starting point is provided below. The primary task is to implement the get_search_points function, which generates a list of search points for a given city:

from src import Gmaps
from src.fields import DEFAULT_FIELDS_WITHOUT_SOCIAL_DATA
from src.write_output import write_output
def uniquify_places_by_id(places):
    """
    Uniquifies a list of dictionaries by 'place_id', preserving the original order.

    :param places: A list of dictionaries, each containing a 'place_id' key among others.
    :return: A list of dictionaries uniquified by 'place_id', preserving the order of first occurrence.
    """
    unique_places = []
    seen_ids = set()  # A set to keep track of seen place_ids for efficient lookups

    for place in places:
        place_id = place.get('place_id')  # Extract the place_id from the current dictionary
        if place_id not in seen_ids:  # If this place_id hasn't been seen before
            unique_places.append(place)  # Add the current dictionary to the result list
            seen_ids.add(place_id)  # Mark this place_id as seen

    return unique_places

def get_search_points(city_name):
    """
    Generates a list of dictionaries containing search points (places along with their coordinates and zoom level)
    for a given city.

    :param city_name: Name of the city for which search points are generated.
    :return: A list of dictionaries with keys 'place', 'coordinates', and 'zoom'.
    """
    # Example search points for demonstration; these would ideally be dynamically generated or fetched from a database
    return [
        {
            "place": "bangalore",
            "geo_coordinates": "12.900490, 77.571466",
            "zoom": 14
        },
        {
            "place": "bangalore",
            "geo_coordinates": "12.906849, 77.577990",
            "zoom": 14
        },
    ]

def fetch_and_print_search_results(search_points):
    """
    Fetches search results for a list of search points and prints the count of results for each point.

    :param search_points: A list of dictionaries, each containing details for a search point.
    """
    # Initialize an empty list to hold all search results
    search_results = []

    # Iterate over each search point to fetch search results
    for point in search_points:
        # Construct the search query for each point
        query = ["hotels in " + point.get("place")]
        zoom_level = point.get("zoom")
        geo_coords = point.get("geo_coordinates")
        
        # Fetch the search results for the current point
        results = Gmaps.places(query, zoom=zoom_level, geo_coordinates=geo_coords)[0]['places']
        
        # Add the fetched results to the overall search_results list
        search_results.extend(results)
        # Print the length of fetched results for the current iteration
        print(f"Retrieved {len(results)} results for {point.get('place')}")

    # Return the collected search results
    return search_results

# Get search points for a specific city
search_points = get_search_points('bangalore')

# The list comprehension iterates over each search point, performs a query, and collects the results
search_results = fetch_and_print_search_results(search_points)

search_results = uniquify_places_by_id(search_results)

write_output("all", search_results, DEFAULT_FIELDS_WITHOUT_SOCIAL_DATA)

[jjwwczy]

https://www.google.com/maps/search/Bus+stops+in+bangalore/@12.9004884,77.571466,14z/data=!3m1!4b1?authuser=0&entry=ttu
https://www.google.com/maps/search/Bus+stops+in+bangalore/@12.906849,77.57799,14z?authuser=0&entry=ttu

Please check the address mannualy.
I tried 2 geo_ordinates in your code, and it returns almost the same search results, is there anything wrong?