Build a secure HUB VPC with Fortigate firewall and alibaba CEN-TR to secure enterprise cross VPCs traffic with terraform
In this tutorial, We will demo how to use Fortigate and Alibaba CEN-TR to secure enterprise east-west and north-south traffic . The term east-west traffic refers to the traffic between two VPCs, VPC-A and VPC-B. The term north-south traffic referes to the traffic from VPC-A or VPC-B outbound to internet. VPC-A and VPC-B itself does not have internet access, instead, it will use fortigate in securehub VPC as the exit to internet. so all traffic can be inspected by Fortigate. Fortigate is located at a VPC named HUBVPC. where it inspect all the traffic from VPC-A /VPC-B. no matter the destination is internet or other VPCs. Fortigate is Next Geneation Firewall, it will able to inspect L3 to L7 traffic. the CEN-TR connect all VPCs together with alicloud internal network to offer a high quality VPC peering network . As all VPCs and CEN-TR in same region. so the traffic between All VPCs are free withouth charge.
For people who are not familar with Fortinet Fortgate firewall and Alibaba CEN-TR , please refer to https://www.fortinet.com/content/dam/fortinet/assets/data-sheets/FortiGate_VM_AliCloud.pdf and https://www.alibabacloud.com/blog/build-an-enterprise-level-private-network-with-cloud-enterprise-network---transit-router-cen-tr_597397
Design
HUBVPC
HUBVPC has 3 vswitches in each AZ zone. one switch called landing vswitch is used for connect CEN-TR, one switch called internal for routing VPC traffic to fortigate, East-West traffic will be routed into HUBVPC and route to fortigate via this internal routing table. the Rest routing table is bound to default vswitch where it has internet access via VPC router. fortigate port 1 is connected into this switch.
Fortigate
Fortigate is a ECS VM in HubVPC, the port1 on fortigate is associated with VPC default switch, so it is the exit to internet, port2 is a ENI on internal vswitch. it will take the traffic from internal vswitch, so the traffic from internal vswitch can arrive fortigate.
VPC-A and VPC-B
VPC-A and VPC-B are the customer workload VPC. it has ECS in both VPCs. the ECS in both VPCs will communicate each other also they will need to access internet. VPC-A and VPC-B only has one default vswitch and default routing table. it route all the traffic to CEN-TR. CEN-TR in turn route all the traffic to HUBVPC for Fortigate to do inspection.
In this tutorial, we can use terraform to create all the resource (you can also use alicloud console instead). after bring up all the resource with all configuration. the ECS in VPCA, VPCB can communication each other and both will able to reach internet. We will verify this by use ping and curl on VM in VPC-A and VPC-B. then we will check fortigate fortivew for traffic activities.
All the script also tested on alibaba computenest service.
Lets begin.
Create CEN
Create a CEN instance
resource "alicloud_cen_instance" "default" {
cen_instance_name = "${var.cen_instance_name}-${random_string.random.result}"
description = "terraform01"
}
a CEN instance created accordingly. the default version is “Enterprise”.
Create CEN-TR
create a TR router in region cn-hongkong associated with cen instance
resource "alicloud_cen_transit_router" "default" {
transit_router_name = "${var.cen_tr_name}-${var.cen_region}-${random_string.random.result}"
cen_id = alicloud_cen_instance.default.id
}
A CEN-TR router is created . by default. it has no Bandwidth Plan associated , however, 1kbps traffic is allowed for testing. which is enough for us to verify the connectivity.
Create CEN-TR routing table
CEN-TR is a virutal router, just like any router can hold routing table, in our tutorial, CEN-TR will require to create 2 routing tables. one routing table is for VPC_A and VPC-B to associate , another routing table is for HUBVPC to associate.
routing table “default_to_hub_for_spoke_vpc” with name “tf-default “ is for VPC_A and VPC_B to associate .
routing table to_spoke_vpc is for hubvpc to associate.
resource "alicloud_cen_transit_router_route_table" "default_to_hub_for_spoke_vpc" {
transit_router_id = alicloud_cen_transit_router.default.transit_router_id
transit_router_route_table_name="${var.cen_tr_name}-${var.cen_tr_table_name_default}"
}
resource "alicloud_cen_transit_router_route_table" "to_spoke_vpc" {
transit_router_id = alicloud_cen_transit_router.default.transit_router_id
transit_router_route_table_name="${var.cen_tr_name}-${var.cen_tr_table_name_to_spoke_vpc}"
}
you will see another default routing table with type “System”. this is default one. in this tutorial. we are not going to use this one. just leave it there. above you shall see two routing table (tf-to_spoke_vpc) and (tf-default) has created.
Attach all VPCs to CEN-TR
As all VPCs in same hongkong region, and we only use one CEN-TR router, so we just attach all VPCs to same CEN-TR router.
Attach hubvpc to CEN-TR
resource "alicloud_cen_transit_router_vpc_attachment" "hubvpc" {
cen_id = alicloud_cen_instance.default.id
transit_router_id = alicloud_cen_transit_router.default.transit_router_id
vpc_id = alicloud_vpc.vpc.id
zone_mappings {
data.alicloud_cen_transit_router_available_resource.default.zones.0.master_zones.0
zone_id = "cn-hongkong-b"
vswitch_id = alicloud_vswitch.landing_for_cen_a_0.id
}
zone_mappings {
data.alicloud_cen_transit_router_available_resource.default.zones.0.slave_zones.0
zone_id = "cn-hongkong-c"
vswitch_id = alicloud_vswitch.landing_for_cen_c_2.id
}
transit_router_attachment_name = "${var.transit_router_attachment_name}-${alicloud_vpc.vpc.name}"
transit_router_attachment_description = "terraform"
}Above we just hard-coded zone b and zone C for TR. if you found your zone for TR is not C and B, please change it accordingly or use zone id data source to get zonemapping.
Attach spoke VPCs to CEN-TR
resource "alicloud_cen_transit_router_vpc_attachment" "spoke_a" {
depends_on =[alicloud_cen_transit_router_vpc_attachment.hubvpc]
cen_id = alicloud_cen_instance.default.id
transit_router_id = alicloud_cen_transit_router.default.transit_router_id
vpc_id = alicloud_vpc.vpc_a.id
zone_mappings {
data.alicloud_cen_transit_router_available_resource.default.zones.0.master_zones.0
zone_id = "cn-hongkong-b"
vswitch_id = alicloud_vswitch.vswitch_a_zone_a.id
}
zone_mappings {
data.alicloud_cen_transit_router_available_resource.default.zones.0.slave_zones.0
zone_id = "cn-hongkong-c"
vswitch_id = alicloud_vswitch.vswitch_a_zone_b.id
}
transit_router_attachment_name = "${var.transit_router_attachment_name}-${alicloud_vpc.vpc_a.name}"
transit_router_attachment_description = "terraform"
}
resource "alicloud_cen_transit_router_vpc_attachment" "spoke_b" {
depends_on = [alicloud_cen_transit_router_vpc_attachment.spoke_a]
cen_id = alicloud_cen_instance.default.id
transit_router_id = alicloud_cen_transit_router.default.transit_router_id
vpc_id = alicloud_vpc.vpc_b.id
zone_mappings {
data.alicloud_cen_transit_router_available_resource.default.zones.0.master_zones.0
zone_id = "cn-hongkong-b"
vswitch_id = alicloud_vswitch.vswitch_b_zone_a.id
}
zone_mappings {
data.alicloud_cen_transit_router_available_resource.default.zones.0.slave_zones.0
zone_id = "cn-hongkong-c"
vswitch_id = alicloud_vswitch.vswitch_b_zone_b.id
}
transit_router_attachment_name = "${var.transit_router_attachment_name}-${alicloud_vpc.vpc_b.name}"
transit_router_attachment_description = "terraform"
}
TR attach to vswitch in both zone B and zone C, so we need to us zone-mapping to map TR zone and VPC zone. here we also use zone B and zone C.
So, far we have cretae CEN,TR,attached (create connection) all VPCs to TR, also created two custom routing table on CEN-TR.
let’s take a look on console for what we have done.
you now have one CEN instance and one CEN-TR router
your TR is attached with 3 VPCs and have 2 cusotme routing table and one system routing table.
Next to do we will need to associate VPCs to TR routing table as well as to populate routing table on CEN-TR.
In our example, traffic from hubvpc will need to use CEN-TR routing table “tf-to-spoke_vpc” , the routing table lookup will be done on this routing table. for traffic that from spoke vpc vpc_a and vpc-b has to use routing table “tf-default” as traffic from vpc-a and vpc-b will consult this routing table to reach fortigate.
to associate hubvpc to routingt able “default_hub_vpc”
resource "alicloud_cen_transit_router_route_table_association" "default_hub_vpc" {
transit_router_route_table_id = alicloud_cen_transit_router_route_table.to_spoke_vpc.transit_router_route_table_id
transit_router_attachment_id = alicloud_cen_transit_router_vpc_attachment.hubvpc.transit_router_attachment_id
}
to associate spoke vpc to routing table “tr-default”
resource "alicloud_cen_transit_router_route_table_association" "associate_spoke_vpc_a_to_tr_rtb_default" {
depends_on =[alicloud_cen_transit_router_route_table_association.associate_spoke_vpc_b_to_tr_rtb_default]
transit_router_route_table_id = alicloud_cen_transit_router_route_table.default_to_hub_for_spoke_vpc.transit_router_route_table_id
transit_router_attachment_id = alicloud_cen_transit_router_vpc_attachment.spoke_a.transit_router_attachment_id
}
resource "alicloud_cen_transit_router_route_table_association" "associate_spoke_vpc_b_to_tr_rtb_default" {
depends_on = [time_sleep.wait_60_seconds]
transit_router_route_table_id = alicloud_cen_transit_router_route_table.default_to_hub_for_spoke_vpc.transit_router_route_table_id
transit_router_attachment_id = alicloud_cen_transit_router_vpc_attachment.spoke_b.transit_router_attachment_id
}
Next We will need to populate CEN-TR routing table.
for tf-to-spoke-vpc. it will require route entry from vpc-a and vpc-b and hub-vpc. so we will need to propogate all VPCs route entries into this routing table.
resource "alicloud_cen_transit_router_route_table_propagation" "default_for_hubvpc_to_spoke_vpc" {
transit_router_route_table_id = alicloud_cen_transit_router_route_table.to_spoke_vpc.transit_router_route_table_id
transit_router_attachment_id = alicloud_cen_transit_router_vpc_attachment.hubvpc.transit_router_attachment_id
}resource "alicloud_cen_transit_router_route_table_propagation" "propagte_spoke_a_vpc_route_to_spoke_vpc_rt" {
transit_router_route_table_id = alicloud_cen_transit_router_route_table.to_spoke_vpc.transit_router_route_table_id
transit_router_attachment_id = alicloud_cen_transit_router_vpc_attachment.spoke_a.transit_router_attachment_id
}
resource "alicloud_cen_transit_router_route_table_propagation" "propagte_spoke_b_vpc_route_to_spoke_vpc_rt" {
transit_router_route_table_id = alicloud_cen_transit_router_route_table.to_spoke_vpc.transit_router_route_table_id
transit_router_attachment_id = alicloud_cen_transit_router_vpc_attachment.spoke_b.transit_router_attachment_id
}
for tf-default. it only need one default route to point to hubvpc via TR. so it does not need to propogate any route into this routing table.
Summary
Now, We have all three VPCs attached to CEN-TR, and two routing table created on this CEN-TR and associated with VPCs , also we propogate route entries from VPCs into TR routing table. so TR know how to get the destination. TR-default routing table does not have any route entries yet as we are not propate any route into this routing table. TR tf-to_spoke_vpc have all the routes propgated from 3 VPCs.
on TR router tf-default routing table. we will need add a static route to tell tr-default all traffic shall be send to hubVPC via TR attachement (to HubVPC).
variable "rt_default" {
default = "0.0.0.0/0"
}
// for config a default route to tr hub-vpc on tf-default for spoke vpc to use
resource "alicloud_cen_transit_router_route_entry" "tr_cen_tf_default_router_entry" {
transit_router_route_table_id = alicloud_cen_transit_router_route_table.default_to_hub_for_spoke_vpc.transit_router_route_table_id
transit_router_route_entry_destination_cidr_block = var.rt_default
transit_router_route_entry_next_hop_type = "Attachment"
transit_router_route_entry_name = "terraform-tf-default-0.0.0.0"
transit_router_route_entry_description = "terraform-tf-default-0.0.0.0"
transit_router_route_entry_next_hop_id = alicloud_cen_transit_router_vpc_attachment.hubvpc.transit_router_attachment_id
}when traffic from spoke vpc reach TR, TR will use tf-default routing table, 0.0.0.0/0 is the only entries in this routing table. so all traffic will be send to hubvpc via Next-hop tr-atta-hubvpc-FG-AP-ljet.
Next We need to config VPCs routing table to tell VPC to send traffic into TR router. for spoke VPC-a and VPC-b which is quite simple. we only need to tell VPC-a and VPC-b to use default route to reach TR.
the terraform script for add route entry in VPC-a and VPC-b
variable "default0" {
default = "0.0.0.0/0"
}
resource "alicloud_route_entry" "spoke_a_default" {
route_table_id = alicloud_vpc.vpc_a.route_table_id
alicloud_route_entry.spoke_vpc_default_route[0].route_table_id
destination_cidrblock = var.default0
nexthop_type = "Attachment"
name = "tf-default_to_tr"
nexthop_id = alicloud_cen_transit_router_vpc_attachment.spoke_a.transit_router_attachment_id
}
resource "alicloud_route_entry" "spoke_b_default" {
alicloud_route_entry.spoke_vpc_default_route[0].route_table_id
route_table_id = alicloud_vpc.vpc_b.route_table_id
destination_cidrblock = var.default0
nexthop_type = "Attachment"
name = "tf-default_to_tr"
nexthop_id = alicloud_cen_transit_router_vpc_attachment.spoke_b.transit_router_attachment_id
}from console, we can config VPC routing table either use VPC menu or in CEN-TR menu, use “Network instace Route Table” menu, here we show how to use “network instance Route table” under CEN-TR , first we filter spoke VPC VPC-A (192.168.10.0/20), add click Add Route Entry to add a static route. do the same for VPC-b.
Do the same for other spoke VPC.
Next we need to config hubvpc routing table.
In hubvpc, things are slightly complicated. As in hubvpc, we have 3 routing table. (2 custom routing table and 1 system default).
one routing table is for fortigate internal network. fortigate port2 is on this network.
System default routing table is just default. it route traffic to internet by default. fortigate port 1 is on this network.
One routing table is for CEN-TR to landing traffic. this routing table will need to route all traffic to port2 ENI ,so all traffic from TR will goes into fortigate.
the landing routing table is associated with landing switch, if the landing switch is connecting with TR via zone B and zone C, then make sure landing switch in zone B and zone C.
let’s create TR landing routing table.
variable "tr_landing_route_entry" {
default = "0.0.0.0/0"
}
resource "alicloud_route_table" "tr_landing_rt" {
depends_on = [time_sleep.wait_120_seconds]
count = 1
vpc_id = alicloud_vpc.vpc.id
route_table_name = "Tr-landing-${random_string.random_name_post.result}-${count.index}"
description = "TR-landing routing table"
}
resource "alicloud_route_entry" "spoke_vpc_default_route" {
count = 1
route_table_id = alicloud_route_table.tr_landing_rt[count.index].id
destination_cidrblock = var.tr_landing_route_entry //Default is 0.0.0.0/0
nexthop_type = "NetworkInterface"
name = alicloud_network_interface.PrimaryFortiGateInterface1.id
nexthop_id = alicloud_network_interface.PrimaryFortiGateInterface1.id
}
resource "alicloud_route_table_attachment" "tr_landing_attachment" {
depends_on = [alicloud_route_table.tr_landing_rt]
count = 1
vswitch_id = alicloud_vswitch.landing_for_cen_a_0.id
route_table_id = alicloud_route_table.tr_landing_rt[count.index].id
}
resource "alicloud_route_table_attachment" "tr_landing_attachment_1" {
depends_on = [alicloud_route_table.tr_landing_rt]
count = 1
vswitch_id = alicloud_vswitch.landing_for_cen_b_1.id
route_table_id = alicloud_route_table.tr_landing_rt[count.index].id
}
resource "alicloud_route_table_attachment" "tr_landing_attachment_2" {
depends_on = [alicloud_route_table.tr_landing_rt]
count = 1
vswitch_id = alicloud_vswitch.landing_for_cen_c_2.id
route_table_id = alicloud_route_table.tr_landing_rt[count.index].id
}
On this hubvpc landing routing table. we will need add a default route (0.0.0.0) to let VPC any traffic coming into this vswitch, lookup tr landing routing table , use default to reach next hop eni port2. so all traffic that from TR will be send to VPC vswitch internal.
variable "tr_landing_route_entry" {
default = "0.0.0.0/0"
}
resource "alicloud_route_table" "tr_landing_rt" {
depends_on = [time_sleep.wait_120_seconds]
count = 1
vpc_id = alicloud_vpc.vpc.id
route_table_name = "Tr-landing-${random_string.random_name_post.result}-${count.index}"
description = "TR-landing routing table"
}
resource "alicloud_route_entry" "spoke_vpc_default_route" {
count = 1
route_table_id = alicloud_route_table.tr_landing_rt[count.index].id
destination_cidrblock = var.tr_landing_route_entry //Default is 0.0.0.0/0
nexthop_type = "NetworkInterface"
name = alicloud_network_interface.PrimaryFortiGateInterface1.id
nexthop_id = alicloud_network_interface.PrimaryFortiGateInterface1.id
}
on hubvpc VPC custom internal routing table. we need to add a static route 192.168.0.0/19 with nexthop to TR. this is for when traffic exit from fortigate to VPC. VPC has to know for traffic that with destination 192.168.0.0/19 , send to TR-attachment towards TR. the nexthop type is forwarding router.
so far, we configured hub VPC route to send traffic into fortigate and handle traffic that leave fortigate . but we have not configured fortigate yet. fortigate take the traffic from tr-landing vswitch and apply security policy and then send it back to VPC vswitch internal.
on fortigate. we have to configurat a static route to let fortigate route traffic that destinated to 192.168.0.0/19 to port2 ,so the traffic can enter TR as port2 is on subnet where it has nexthop TR to reach 192.168.0.0/19.
so let’s configure fortigate. on fortigate we need configure both routing table and firewall policy.
above are the all configuration.
Let’s verify the result.
ssh into fortigate, then use “execute ssh 192.168.10.100” to login into VM in spoke VPC-A. the VM shall able to reach it’s VPC-B router (192.168.20.253) and VM in VPC-B (192.168.20.100).
Because traffic from VPC-A VM to VPC-B VM is via Fortigate via Port2, so
From Fortigate Fortiview, you will able to see the traffic between two VMs in different VPCs.
Config Firewall policy on fortigate for North-South traffic.
For VPC-A and VPC-B internet bound traffic. VM on VPC-A and VPC-B does not have local internet access. it only has default route to CEN-TR, CEN-TR will lookup TF-default routing table and send traffic to hubvpc. hubvpc will lookup TR-landing routing table and send traffic to fortigate port2 eni . so traffic arrive fortigate. once traffic arrive fortigate. fortigate will lookup the routing table , it will find that to internet the nexthop is port1 on fortigate. so next-hop , the firewall policy will be checked to see whether this is allowed. obviously, we have not configured firewall policy yet. so let’s config it.
config firewall policy
edit 2 set name "port2_to_port1"
set srcintf "port2"
set dstintf "port1"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ALL"
set utm-status enable
set ssl-ssh-profile "certificate-inspection"
set av-profile "default"
set webfilter-profile "default"
set ips-sensor "default"
set application-list "default"
set logtraffic all
set nat enable
We have to enable NAT, and we enable UTM feature as well. so the traffic from VM in VPC-A and VPC-B will be insepcted by fortigate.
for example, when I issue curl -I www.google.com/ls.exe on VM in VPA-A
the fortigate will block this as this is unrated traffic according fortigate UTM policy.
Summary
Above we showed how we use Fortigate to inspect traffic between VPCs and VPC to internetl, Alibaba CEN-TR function as a virtual router and connect all VPCs together and routing traffic between VPCs.
This demo case has also avaiable on alibaba computenest platform. you can go and try them and find all the script as well.
