Python web scrapping on GCP

Content

  1. build container images.
  2. tag images.
  3. upload images to GCP.
  4. create services.

build container images.

  • selenium.
  • web scrapping.

tag images.

  • image with GCP pattern. 
docker pull selenium/standalone-chrome
# tag with pattern $GCP_REGISTRY/$PROJECT_NAME/$IMAGE_NAME
docker tag selenium_standalone-chrome asia.gcr.io/$PROJECT_NAME/selenium-standalone-chrome
# push image
docker push asia.gcr.io/$PROJECT_NAME/selenium-standalone-chrome

upload images to GCP.

  • authentication on browser. 
gcloud auth login
  • setup project. 
gcloud config set project $PROJECT_NAME

create services.

  • Create from container images. 
# create selenium service.
gcloud run deploy selenium-chrome --image asia.gcr.io/$PROJECT_NAME/selenium_standalone-chrome --port 4444 --memory 2G --region asia-southeast1 --platform managed
  • Create from cloud build. 
# gcloud builds submit --tag asia.gcr.io/$PROJECT_NAME/$IMAGE_NAME

gcloud builds submit --tag asia.gcr.io/$PROJECT_NAME/web-scrpping
# deploy service
gcloud run deploy selenium-chrome --image asia.gcr.io/$PROJECT_NAME/selenium_standalone-chrome

Reference:
https://www.roelpeters.be/how-to-deploy-a-scraping-script-and-selenium-in-google-cloud-run/
https://dev.to/googlecloud/using-headless-chrome-with-cloud-run-3fdp

43

Categories:

Python

Tags:

PythonSeleniumGcp