Write script to scrap house rent price and send to your phone
House rental is a critical thing if you are living in the silicon valley. I'm planning to move to a new apartment, but the price changes often. I want to monitor the price and book it when it drops. In order to do that, I wrote a script to monitor the house rental price and push the result to my phone like below.
Get Started
The apartment I'm looking at is a professional rental apartment, to not disclose the place I will be living in, I'm using another one for example.
The official website of the apartment is:
https://www.avaloncommunities.com/california/mountain-view-apartments/avalon-mountain-view/floor-plans
The pricing section:
Analysis
To monitor the price, the script will need to:
- Get the price, size, move-in-date, and other interesting information
- Compose to a message and send to me(I personally use Telegram Bot, refer to:Use Telegram and Flask to build personal notification server)
Implementation
Originally, I plan to use spider to scrap html text and extract information, however, I discovered that the websiste use public REST API as interface for front-end client. To make sure the API works without authentication/authorization, I copied the GET request to an anonymous window and it works fine. So this API can be used to get all information I need.
Chrome Inspector:
After further research, I found that this API even support additional parameters to set minimum price, maximum price, and move-in date, etc.
Now it's time to implement, as I have a Flask app running on Google App Engine, I will be writing Python code to do the job. The job will be triggered by a HTTP request to the Flask app. You can use GitLab to make the HTTP call to the Flask app as introduced here Use GitLab-CI to run free scheduled jobs
Code
Flask App
@app.route('/housing', methods=['GET'])
def housing():
houseResponse = json.loads(urlfetch.fetch(url='https://api.avalonbay.com/json/reply/ApartmentSearch?communityCode=CA049&min=2000&max=3000&desiredMoveInDate=2018-09-01T07:00:00.000Z').content)
availableFloorPlans = houseResponse["results"]["availableFloorPlanTypes"][0]["availableFloorPlans"]
responseText = "*House Quotes:*%0A"
for availableFloorPlan in availableFloorPlans:
for apartment in availableFloorPlan["finishPackages"][0]["apartments"]:
responseText += "Size:" + str(apartment["apartmentSize"]) + " Price:" + str(apartment["pricing"]["effectiveRent"]) + " Floor:" + str(apartment["floor"]) + " Date:" + time.strftime('%Y-%m-%d', time.localtime(float(filter(str.isdigit, str(apartment["pricing"]["availableDate"])))/1000)) + "%0A"
requestUrl = config.telegramWebHookURI + "/sendMessage?parse_mode=Markdown&chat_id=123456789" + "&text=" + responseText
urlfetch.fetch(url=requestUrl).content
return responseText
2
3
4
5
6
7
8
9
10
11
GitLab-CI
GitLab-CI Configuration (gitlab-ci.yml):
test:
script:
- bash scripts/get_quotes.sh
2
3
GitLab-CI Script (get_quotes.sh):
#!/bin/sh
curl https://api.tonyxu.io/housing
2
Demo
