Write script to scrap house rent price and send to your phone
House rental is a critical thing if you are living in the silicon valley. I'm planning to move to a new apartment, but the price changes often. I want to monitor the price and book it when it drops. In order to do that, I wrote a script to monitor the house rental price and push the result to my phone like below.
The apartment I'm looking at is a professional rental apartment, to not disclose the place I will be living in, I'm using another one for example.
The official website of the apartment is:
The pricing section:
To monitor the price, the script will need to:
- Get the price, size, move-in-date, and other interesting information
- Compose to a message and send to me(I personally use Telegram Bot, refer to:Use Telegram and Flask to build personal notification server)
Originally, I plan to use spider to scrap html text and extract information, however, I discovered that the websiste use public REST API as interface for front-end client. To make sure the API works without authentication/authorization, I copied the GET request to an anonymous window and it works fine. So this API can be used to get all information I need.
After further research, I found that this API even support additional parameters to set minimum price, maximum price, and move-in date, etc.
Now it's time to implement, as I have a Flask app running on Google App Engine, I will be writing Python code to do the job. The job will be triggered by a HTTP request to the Flask app. You can use GitLab to make the HTTP call to the Flask app as introduced here Use GitLab-CI to run free scheduled jobs
@app.route('/housing', methods=['GET']) def housing(): houseResponse = json.loads(urlfetch.fetch(url='https://api.avalonbay.com/json/reply/ApartmentSearch?communityCode=CA049&min=2000&max=3000&desiredMoveInDate=2018-09-01T07:00:00.000Z').content) availableFloorPlans = houseResponse["results"]["availableFloorPlanTypes"]["availableFloorPlans"] responseText = "*House Quotes:*%0A" for availableFloorPlan in availableFloorPlans: for apartment in availableFloorPlan["finishPackages"]["apartments"]: responseText += "Size:" + str(apartment["apartmentSize"]) + " Price:" + str(apartment["pricing"]["effectiveRent"]) + " Floor:" + str(apartment["floor"]) + " Date:" + time.strftime('%Y-%m-%d', time.localtime(float(filter(str.isdigit, str(apartment["pricing"]["availableDate"])))/1000)) + "%0A" requestUrl = config.telegramWebHookURI + "/sendMessage?parse_mode=Markdown&chat_id=123456789" + "&text=" + responseText urlfetch.fetch(url=requestUrl).content return responseText
GitLab-CI Configuration (gitlab-ci.yml):
test: script: - bash scripts/get_quotes.sh
GitLab-CI Script (get_quotes.sh):
#!/bin/sh curl https://api.tonyxu.io/housing