Backup mongodb to GCP storage with crontab

情況是要把原本寫入 local mongodb 的 usage log dump 出 csv 檔案(並且從mongodb內刪除)，並且把較舊的 csv 檔案移去 GCP storage。

mongodump.sh

#!/usr/bin/env bash
MY_DATE="'$(date +"%Y-%m-%d")'"
#echo $MY_DATE
MY_DATE_QUERY="{'recorddatetime': {\$lt: ${MY_DATE} } }"
#echo $MY_DATE_QUERY

#backup
echo "$(mongoexport -d transdb -c validtrans --query " ${MY_DATE_QUERY} " --fields=fieldA,fieldB,fieldC,fieldD,recorddatetime --quiet --out=/var/opt/mongodump/validtrans_$(date "+%Y%m%d").csv --type=csv)"

#delete
echo "$(mongo transdb --quiet --eval "db.validtrans.deleteMany( ${MY_DATE_QUERY} ) " )"


#move to bucket
echo "$(find /var/opt/mongodump/*.csv -type f -ctime +30 -exec gsutil mv {} gs://backup_bucket/path_to_destination/ \;)"

實際上執行的時候會有錯誤訊息

$ bash ./dump_mongo_keyusage.sh 
Copying file:///var/opt/mongodump/validtrans_20220131.csv [Content-Type=text/csv]...
Your "Oauth 2.0 User Account" credentials are invalid. Please run
  $ gcloud auth login
Failure: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590).

除了做 gcloud auth login 之外，還要做 gcloud auth activate-service-account。詳細可以參考 https://cloud.google.com/sdk/gcloud/reference/auth/activate-service-account。同時要搭配 key json file
，會長得類似這樣…

{
  "type": "service_account",
  "project_id": "my-services",
  "private_key_id": "PRIVATE_KEY_ID",
  "private_key": "-----BEGIN PRIVATE KEY-----\nPRIVATE_KEY==\n-----END PRIVATE KEY-----\n",
  "client_email": "[email protected]",
  "client_id": "",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/my-services%40appspot.gserviceaccount.com"
}

執行 gcloud auth activate-service-account

$ gcloud auth activate-service-account [email protected] --key-file=/var/opt/key.json --project=my-services

然後執行 mongodump.sh 就可以了

$ ./mongodump.sh
Uploading   gs://keyusage_raw_backup/validtrans_20220131.csv:   106.39 MiB/106.39 MiB    
Removing file:///var/opt/mongodump/validtrans_20220131.csv...
Copying file:///var/opt/mongodump/validtrans_20220131.csv [Content-Type=text/csv]...

Original link: Phanix's Blog