Background
寫這篇文章的目的就是想要之後我送了一個PR出去通過circleCI的檢查後, 然後就會小精靈會幫我自動更新程式到我指定的機器. 但事實是沒有小精靈, 所以過往我只完成到 circleCI檢查的部分. 確定都沒問題之後我才會手動的更新程式 & 重啟服務這樣. 那今天終於把整個流程實作了出來, 所以就寫了這篇文章記錄一下.
整個 Deployment 的流程如下:
這篇文章順便整理一下對 CircleCI 的應用. 比較基礎的部分我就跳過了, 基本上就創造一個CircleCI & Github 的帳號, 再設定一下兩邊基本上就通了. 所以本篇主要在介紹 CircleCI 的 config 部分. 我們會透過 CircleCI 的 config 部分來完成 aws codeploy 的呼叫.
CircleCI configuration
Concept
整個 CircleCI configuration file 由三大部分所組成. 分別是:
- version: version表示你要用CI的哪個版本. 目前是2.0
- Jobs: 一系列你想要在 CircleCI 的機器上操作的指令. 通常包含建立環境 & Testing
- workflows: 用來管理Jobs的部分, 如果想要run不同的jobs在不同的branchs上就是透過workflows來完成.
Basic
基本的 basic CircleCI config 的架構如下:
version: 2
# List of jobs
jobs:
# The build job
build:
working_directory: ~/project
docker:
- image: circleci/node:6.10-browsers
steps:
# Steps to be added
# The deploy job
deploy:
working_directory: ~/project
docker:
- image: circleci/node:6.10-browsers
steps:
# Steps to be added
workflows:
version: 2
# The build and deploy workflow
build_and_deploy:
jobs:
- build
- deploy
如上圖所示有兩個jobs(build, deploy). 那jobs裡面的細項如下:
- docker
- working_directory
- steps
- checkout
- run
- save_cache
- restore_cache
需要特別講的部分是, 在 workflow 中, 每個 job 都是運作在一個獨立的 container 當中, 爲了確保每個 job 可以獨立運行, 所以可能會需要前一次 CI 或是前一個 job 產生的一些資料, 這時候我們就會用到 cache 來存放資料.
名詞解釋
- docker: 指定你要用什麼 docker image
- working_directory: 指定工作目錄
- steps: 一連串的動作來幫助你完成CI/CD
- run: 執行 bash command.
- checkout: checkout the code from the current branch into the working directory
- save_cache: 儲存你要存的資料到 cache 中, 透過key-valuek的形式.
- restore_cache: 取回你存的資料, 透過key-valuek的形式.
AWS Codedeploy
簡單地介紹完 CicleCI 後我們還會需要設定 AWS Codedeploy.
Add a new user
我們創建了一個 AWS IAM user 叫 circleci 並 attach 相關的 policy 給這個 user. 以下是相關的 policy:
AmazonS3FullAccess
AWSCodeDeployFullAccess
Launch EC2 Instance
我們要先叫起一台 EC2 然後在上面安裝 aws cli 等相關指令. 但是在創造 EC2 時要指定 IAM Role 選擇系統預設的 CodeDeployEC2
(如果沒有的話需自行建立)
相關policy如下:
AmazonEC2FullAccess
AmazonEC2RoleforAWSCodeDeploy
AWSCodeDeployRole
AWSCodeDeployDeployerAccess
下面的指令你可以在創建EC2的時候填入(User data),或是等EC2創建好時在登入執行都可以。
$ sudo apt-get update
$ sudo apt-get install python-pip
$ sudo pip install awscli --upgrade --user
$ aws --version
aws-cli/1.16.24 Python/2.7.10 Darwin/18.2.0 botocore/1.12.14
$ aws s3 cp s3://aws-codedeploy-us-west-1/latest/install .
$ chmod +x ./install
$ ./install auto
P.S 記得確定一下 codedeploy-agent 服務在 instance 上有沒有啟動
sudo service codedeploy-agent status
● codedeploy-agent.service - LSB: AWS CodeDeploy Host Agent
Loaded: loaded (/etc/init.d/codedeploy-agent; generated)
Active: active (exited) since Tue 2019-05-07 13:55:51 CST; 1s ago
Docs: man:systemd-sysv-generator(8)
Process: 12433 ExecStart=/etc/init.d/codedeploy-agent start (code=exited, status=0/SUCCESS)
May 07 13:55:50 ip-172-31-18-88 systemd[1]: Starting LSB: AWS CodeDeploy Host Agent...
May 07 13:55:51 ip-172-31-18-88 codedeploy-agent[12433]: Starting codedeploy-agent:
May 07 13:55:51 ip-172-31-18-88 systemd[1]: Started LSB: AWS CodeDeploy Host Agent.
Codedeploy Application
到 Codedeploy 中創立一個 application, 如下圖:
Deployment groups
接著我們再創立一個 deployment groups 把我們剛剛建立好的 EC2 包含進來
Load Balancer 則看你有沒有啟用,有的話就選擇 Target Group,沒有則取消,Advanced 的選項中有SNS觸發的選項,通常我只針對Successful & Fail這兩種狀況進行監控。
appspec.yml
那接著如果我們要使用 aws codedeploy 時我們需要寫 appspec.yml
來告訴aws codedeploy需要執行哪些動作 & 基本的路徑位置. 在appspec.yml中我們比較需要的是填入三個script(BeforeInstall, AfterInstall, ApplicationStart). 那我自己這邊是用不到 BeforeInstall 所以我就把整個欄位拿掉.
appspec.yml
version: 0.0
os: linux
files:
- source: /
destination: /var/www/app
hooks:
AfterInstall:
- location: scripts/after-install.sh
runas: root
timeout: 300
ApplicationStart:
- location: scripts/restart.sh
runas: root
timeout: 60
scripts/after-install.sh
#!/bin/bash
cd /var/www/app
source .venv/bin/activate
pip3 install -r requirements.txt
scripts/restart.sh
#!/bin/bash
sudo supervisorctl restart app
整合 Circleci & Aws CodeDeploy
最後我們要透過 circleci 的 config 整合 CodeDeploy 來達到自動部署機器的目的.
config.yml
version: 2
jobs:
build:
working_directory: ~/app
docker:
- image: circleci/python:3.6.4
environment:
PIPENV_VENV_IN_PROJECT: true
DATABASE_URL: postgresql://root@localhost/circle_test?sslmode=disable
- image: circleci/postgres:9.6.2
environment:
POSTGRES_USER: taiker
POSTGRES_DB: console
steps:
- checkout
- run: sudo apt-get install python3-pip
- run: sudo pip3 install --upgrade pip
- run:
command: |
python3 -m venv venv
. venv/bin/activate
sudo pip3 install -r requirements.txt
- save_cache:
key: deps1-{{ .Branch }}-{{ checksum "requirements.txt" }}
paths:
- venv
- run:
command: |
export FLASK_APP=/home/circleci/app/run.py
pytest
deploy:
working_directory: ~/app
docker:
- image: circleci/python:3.6.4
steps:
- checkout
- restore_cache:
key: deps1-{{ .Branch }}-{{ checksum "requirements.txt" }}
- run:
name: Show current branch
command: echo ${CIRCLE_BRANCH}
- run:
name: Show git commit id
command: echo ${CIRCLE_SHA1}
- run: sudo apt-get update
- run: sudo apt-get install python3-pip
- run: sudo apt-get install groff
- run: sudo pip3 install --upgrade pip
- run:
name: Install aws cli
command: pip3 install awscli --upgrade --user
- run: ~/.local/bin/aws --version
- run: chmod +x ~/.local/bin/aws
- run:
name: "Set AWS region"
command: ~/.local/bin/aws configure set region ap-northeast-1
- run:
name: Deploy to S3
command: |
if [ "${CIRCLE_BRANCH}" == "qa" ]; then
~/.local/bin/aws s3 sync /home/circleci/app/ s3://app/qa --delete
elif [ "${CIRCLE_BRANCH}" == "master" ]; then
~/.local/bin/aws s3 sync /home/circleci/app/ s3://app/master --delete
fi
- run:
name: AWS codedeploy
command: |
if [ "${CIRCLE_BRANCH}" == "qa" ]; then
~/.local/bin/aws deploy create-deployment --application-name app-qa --deployment-group-name app-group-qa --file-exists-behavior OVERWRITE --github-location repository="TaikerLiang/APP",commitId="$CIRCLE_SHA1"
elif [ "${CIRCLE_BRANCH}" == "master" ]; then
~/.local/bin/aws deploy create-deployment --application-name app --deployment-group-name app-group --file-exists-behavior OVERWRITE --github-location repository="TaikerLiang/APP",commitId="$CIRCLE_SHA1"
fi
workflows:
version: 2
# The build and deploy workflow
build_and_deploy:
jobs:
- build
# The deploy job will only run on the filtered branches and
# require the build job to be successful before it starts
- deploy:
requires:
- build
filters:
branches:
only:
- master, qa
那整篇文章大致上我覺得比較重要的部分都列在這邊. 感覺可能還有需要補充的地方. 等之後比較有空的時候再來補充.
Reference
- https://medium.freecodecamp.org/our-journey-for-using-circleci-2-0-to-build-and-deploy-an-angular-app-to-aws-s3-8e7ea3f51503
- https://medium.com/reactbrasil/continuous-integration-e-delivery-com-circleci-2-0-86d137ba1826
- https://medium.com/statementdog-engineering/boost-ci-build-with-circleci-workflow-89474162ca3a
- https://devalon.biz/circlecikaraaws-codedeploy-ripozitoritaipu-github-woshi-xing-suru/
- https://oranwind.org/-aws-windows-chuan-song-dang-an-dao-aws-ec2-jiao-xue-3/
- https://www.blog.labouardy.com/continuous-deployment-with-aws-codedeploy-github/