My thoughts of life and code

Write script to scrap house rent price and send to your phone

Tony Xu


House rental is a critical thing if you are living in the silicon valley. I’m planning to move to a new apartment, but the price changes often. I want to monitor the price and book it when it drops. In order to do that, I wrote a script to monitor the house rental price and push the result to my phone like below.

Get Started

The apartment I’m looking at is a professional rental apartment, to not disclose the place I will be living in, I’m using another one for example.

The official website of the apartment is:

The pricing section:


To monitor the price, the script will need to:

  1. Get the price, size, move-in-date, and other interesting information
  2. Compose to a message and send to me(I personally use Telegram Bot, refer to:Use Telegram and Flask to build personal notification server)


Originally, I plan to use spider to scrap html text and extract information, however, I discovered that the websiste use public REST API as interface for front-end client. To make sure the API works without authentication/authorization, I copied the GET request to an anonymous window and it works fine. So this API can be used to get all information I need.

Chrome Inspector:

After further research, I found that this API even support additional parameters to set minimum price, maximum price, and move-in date, etc.

Now it’s time to implement, as I have a Flask app running on Google App Engine, I will be writing Python code to do the job. The job will be triggered by a HTTP request to the Flask app. You can use GitLab to make the HTTP call to the Flask app as introduced here Use GitLab-CI to run free scheduled jobs


Flask App

@app.route('/housing', methods=['GET'])
def housing():
    houseResponse = json.loads(urlfetch.fetch(url='').content)
    availableFloorPlans = houseResponse["results"]["availableFloorPlanTypes"][0]["availableFloorPlans"]
    responseText = "*House Quotes:*%0A"
    for availableFloorPlan in availableFloorPlans:
        for apartment in availableFloorPlan["finishPackages"][0]["apartments"]:
            responseText += "Size:" + str(apartment["apartmentSize"]) + " Price:" + str(apartment["pricing"]["effectiveRent"]) + " Floor:" + str(apartment["floor"]) + " Date:" + time.strftime('%Y-%m-%d', time.localtime(float(filter(str.isdigit, str(apartment["pricing"]["availableDate"])))/1000)) + "%0A"
    requestUrl = config.telegramWebHookURI + "/sendMessage?parse_mode=Markdown&chat_id=123456789" + "&text=" + responseText
    return responseText


GitLab-CI Configuration (gitlab-ci.yml):

    - bash scripts/

GitLab-CI Script (



Subscribe to get update about programming, hacks, and other topics.

comments powered by Disqus