API requests with a For Loop

If I needed to make get requests to an endpoint that has a page limit (50 items) and a total items limit of 1000:

I could use a for loop to build a new URL every time by concatenating the base URL + page limit + offset in order to get all items. However, I wouldn’t be using the parameter “parameters” in the get request.

The page limit is always the same. Should I be using a parameters dictionary with ‘limit’: ‘50’ and only use the url concatenation to include the offset? Or is it OK to avoid using the parameters dictionary and include the page limit im the URL concatenation?

Using range, range(start, end):

for i in range(start, end):

Or using slicing

for i in H[start:end+1]:

Using range, range(start, end, step):
Increment by a certain step at each iteration

for i in range(start, end, step):

Or using slicing with step

for i in H[start:end+1, step]:

OR using while loop

H = [1,2,3,7,9,0}
i = start 
n = len(H)
while i < n:
    i += 1

Hi Yanina,

I wrote a script not long ago to extract used-car ads from a website by querying an unofficial API endpoint. Maybe by looking at my code you will find the answer you are looking for. Feel free to ask any questions.

import requests
import json
import time

url = "https://api.leboncoin.fr/finder/search"
brand = "Peugeot"
model = "308"

ads = [] # this is where I am going to store the data

for i in range(0,5200,100): # This is the for-loop for constructing the post-requests 
# (yes, this is weird. Normally you would use a get request for retrieving info from a 
# website but this is just how this API endpoint is designed) I want to get 5200 total ads 
# with an offset of 100.

    payload = {"filters":{"category":{"id":"2"},"enums":{"ad_type":["offer"], "brand": [brand], "model": [model]},"keywords":{"text":"peugeot 308 sw"},"location":{},"ranges":{}},"limit":100,"limit_alu":0, "offset": i}  

# Take a look at the last parameter in the payload dictionary. This is where the looping variable is found.
# The rest of the code appends the data (sweet, sweet json!) obtained in each post 
# request to the list ads. Once the foor-loop is done and all the ads are downloaded,  
# the data is saved to a json file for later processing.

    response = requests.post(url, json=payload)
    status = response.status_code

    data = json.loads(response.text)
    ads.append(data['ads'])
    time.sleep(3) 
# This is just me trying to be nice and not overloading their servers (or get myself 
# banned) by sending to many requests at once. time.sleep(3) introduces a #3-second 
# delay between requests.

with open("raw_ads.json", "w") as f:
    f.write(json.dumps(ads))