Monday, October 23, 2023
HomeBig DataResolve non-public DNS hostnames for Amazon MSK Join

Resolve non-public DNS hostnames for Amazon MSK Join

Amazon MSK Join is a characteristic of Amazon Managed Streaming for Apache Kafka (Amazon MSK) that provides a totally managed Apache Kafka Join surroundings on AWS. With MSK Join, you may deploy absolutely managed connectors constructed for Kafka Join that transfer knowledge into or pull knowledge from widespread knowledge shops like Amazon S3 and Amazon OpenSearch Service. With the introduction of the Personal DNS help into MSK Join, connectors are capable of resolve non-public buyer domains, utilizing their DNS servers configured within the buyer VPC DHCP Choices set. This submit demonstrates an answer for resolving non-public DNS hostnames outlined in a buyer VPC for MSK Join.

Chances are you’ll wish to use non-public DNS hostname help for MSK Join for a number of causes. Earlier than the non-public DNS decision functionality included with MSK Join, it used the service VPC DNS resolver for DNS decision. MSK Join didn’t use the non-public DNS servers outlined within the buyer VPC DHCP choice units for DNS decision. The connectors have been solely capable of reference hostnames within the connector configuration or plugin which are publicly resolvable and couldn’t resolve non-public hostnames outlined in both a non-public hosted zone or use DNS servers in one other buyer community.

Many purchasers be sure that their inside DNS purposes are usually not publicly resolvable. For instance, you might need a MySQL or PostgreSQL database and should not need the DNS identify on your database to be publicly resolvable or accessible. Amazon Relational Database Service (Amazon RDS) or Amazon Aurora servers have DNS names which are publicly resolvable however not accessible. You’ll be able to have a number of inside purposes corresponding to databases, knowledge warehouses, or different methods the place DNS names are usually not publicly resolvable.

With the current launch of MSK Join non-public DNS help, you may configure connectors to reference public or non-public domains. Connectors use the DNS servers configured in your VPC’s DHCP choice set to resolve domains. Now you can use MSK Connect with privately join with databases, knowledge warehouses, and different sources in your VPC to conform along with your safety wants.

In case you have a MySQL or PostgreSQL database with non-public DNS, you may configure it on a customized DNS server and configure the VPC-specific DHCP choice set to do the DNS decision utilizing the customized DNS server native to the VPC as a substitute of utilizing the service DNS decision.

Resolution overview

A buyer can have completely different structure choices to arrange their MSK Join. For instance, they will have Amazon MSK and MSK Join are in the identical VPC or supply system in VPC1 and Amazon MSK and MSK Join are in VPC2 or supply system, Amazon MSK and MSK Join are all in several VPCs.

The next setup makes use of two completely different VPCs, the place the MySQL VPC hosts the MySQL database and the MSK VPC hosts Amazon MSK, MSK Join, the DNS server, and numerous different elements. You’ll be able to lengthen this structure to help different deployment topologies utilizing acceptable AWS Identification and Entry Administration (IAM) permissions and connectivity choices.

This submit gives step-by-step directions to arrange MSK Join the place it can obtain knowledge from a supply MySQL database with non-public DNS hostname within the MySQL VPC and ship knowledge to Amazon MSK utilizing MSK Join in one other VPC. The next diagram illustrates the high-level structure.

The setup directions embody the next key steps:

  1. Arrange the VPCs, subnets, and different core infrastructure elements.
  2. Set up and configure the DNS server.
  3. Add the information to the MySQL database.
  4. Deploy Amazon MSK and MSK Join and eat the change knowledge seize (CDC) data.


To comply with the tutorial on this submit, you want the next:

Create the required infrastructure utilizing AWS CloudFormation

Earlier than configuring the MSK Join, we have to arrange the VPCs, subnets, and different core infrastructure elements. To arrange sources in your AWS account, full the next steps:

  1. Select Launch Stack to launch the stack in a Area that helps Amazon MSK and MSK Join.
  2. Specify the non-public key that you just use to connect with the EC2 situations.
  3. Replace the SSH location along with your native IP tackle and preserve the opposite values as default.
  4. Select Subsequent.
  5. Evaluation the main points on the ultimate web page and choose I acknowledge that AWS CloudFormation would possibly create IAM sources.
  6. Select Create stack and anticipate the required sources to get created.

The CloudFormation template creates the next key sources in your account:

  • VPCs:
  • Subnets within the MSK VPC:
    • Three non-public subnets for Amazon MSK
    • Personal subnet for DNS server
    • Personal subnet for MSKClient
    • Public subnet for bastion host
  • Subnets within the MySQL VPC:
    • Personal subnet for MySQL database
    • Public subnet for bastion host
  • Web gateway hooked up to the MySQL VPC and MSK VPC
  • NAT gateways hooked up to MySQL public subnet and MSK public subnet
  • Route tables to help the visitors stream between completely different subnets in a VPC and throughout VPCs
  • Peering connection between the MySQL VPC and MSK VPC
  • MySQL database and configurations
  • DNS server
  • MSK shopper with respective libraries

Please notice, for those who’re utilizing VPC peering or AWS Transit Gateway with MSK Join, don’t configure your connector for reaching the peered VPC sources with IPs within the CIDR ranges. For extra info, check with Connecting from connectors.

Configure the DNS server

Full the next steps to configure the DNS server:

  1. Connect with the DNS server. There are three configuration recordsdata out there on the DNS server underneath the /residence/ec2-user folder:
    • named.conf
  2. Run the next instructions to put in and configure your DNS server:
    sudo yum set up bind bind-utils –y
    cp /residence/ec2-user/named.conf /and many others/named.conf
    chmod 644 /and many others/named.conf
    cp /var/named/
    cp /var/named/

  3. Replace /and many others/named.conf.

For the allow-transfer attribute, replace the DNS server inside IP tackle to allow-transfer

{ localhost; <DNS Server inside IP tackle>; };.

You could find the DNS server IP tackle on the CloudFormation template Outputs tab.

Word that the MSK cluster continues to be not arrange at this stage. We have to replace the Kafka dealer DNS names and their respective inside IP addresses within the /var/named/ configuration file after organising the MSK cluster later on this submit. For directions, check with right here.

Additionally notice that these settings configure the DNS server for this submit. In your personal surroundings, you may configure the DNS server as per your wants.

  1. Restart the DNS service:
    sudo su
    service named restart

You need to see the next message:

Redirecting to /bin/systemctl restart named.service

Your customized DNS server is up and working now.

Add the information to the MySQL database

Usually, we are able to use an Amazon RDS for MySQL database, however for this submit, we use customized MySQL database servers. The Amazon RDS DNS is publicly accessible and MSK Join helps it, nevertheless it was not capable of help databases or purposes with non-public DNS up to now. With the newest non-public DNS hostnames characteristic launch, it might help purposes’ non-public DNS as nicely, so we use a MySQL database on the EC2 occasion.

This set up gives details about organising the MySQL database on a single-node EC2 occasion. This shouldn’t be used on your manufacturing setup. You need to comply with acceptable steerage for organising and configuring MySQL in your account.

The MySQL database is already arrange utilizing the CloudFormation template and is able to use now. To add the information, full the followings steps:

  1. SSH to the MySQL EC2 occasion. For directions, check with Connect with your Linux occasion. The information file salesdb.sql is already downloaded and out there underneath the /residence/ec2-user listing.
  2. Log in to mysqldb with the person identify grasp.
  3. To entry the password, navigate to AWS Programs Supervisor and Parameter Retailer tab. Choose /Database/Credentials/grasp and click on on View Particulars and replica the worth for the important thing.
  4. Log in to MySQL utilizing the next command:
    mysql -umaster -p<MySQLMasterUserPassword>

  5. Run the next instructions to create the salesdb database and cargo the information to the desk:
    use salesdb;
    supply /residence/ec2-user/salesdb.sql;

This can insert the data in numerous completely different tables within the salesdb database.

  1. Run present tables to see the next tables within the salesdb:
    mysql> present tables;
    | Tables_in_salesdb |
    | CUSTOMER |
    | PRODUCT |
    | SUPPLIER |

Create a DHCP choice set

DHCP choice units offer you management over the next elements of routing in your digital community:

  • You’ll be able to management the DNS servers, domains, or Community Time Protocol (NTP) servers utilized by the gadgets in your VPC.
  • You’ll be able to disable DNS decision utterly in your VPC.

To help non-public DNS, you should use an Amazon Route 53 non-public zone or your personal customized DNS server. In case you use a Route 53 non-public zone, the setup will work mechanically and there’s no must make any modifications to the default DHCP choice set for the MSK VPC. For a customized DNS server, full the next steps to arrange a customized DHCP configuration utilizing Amazon Digital Personal Cloud (Amazon VPC) and fasten it to the MSK VPC.

There might be a default DHCP choice set in your VPC hooked up to the Amazon supplied DNS server. At this stage, the requests will go to Amazon’s supplied DNS server for decision. Nonetheless, we create a brand new DHCP choice set as a result of we’re utilizing a customized DNS server.

  1. On the Amazon VPC console, select DHCP choice set within the navigation pane.
  2. Select Create DHCP choice set.
  3. For DHCP choice set identify, enter MSKConnect_Private_DHCP_OptionSet.
  4. For Area identify, enter mysql.inside.
  5. For Area identify server, enter the DNS server IP tackle.
  6. Select Create DHCP choice set.
  7. Navigate to the MSK VPC and on the Actions menu, select Edit VPC settings.
  8. Choose the newly created DHCP choice set and reserve it.
    The next screenshot reveals the instance configurations.
  9. On the Amazon EC2 console, navigate to privateDNS_bastion_host.
  10. Select Occasion state and Reboot occasion.
  11. Wait a couple of minutes after which run nslookup from the bastion host; it ought to have the ability to resolve it utilizing your native DNS server as a substitute of Route 53:
nslookup native.mysql.inside

Now our base infrastructure setup is able to transfer to the subsequent stage. As a part of our base infrastructure, we have now arrange the next key elements efficiently:

  • MSK and MySQL VPCs
  • Subnets
  • EC2 situations
  • VPC peering
  • Route tables
  • NAT gateways and web gateways
  • DNS server and configuration
  • Applicable safety teams and NACLs
  • MySQL database with the required knowledge

At this stage, the MySQL DB DNS identify is resolvable utilizing a customized DNS server as a substitute of Route 53.

Arrange the MSK cluster and MSK Join

The following step is to deploy the MSK cluster and MSK Join, which is able to fetch data from the salesdb and ship it to an Amazon Easy Storage Service (Amazon S3) bucket. On this part, we offer a walkthrough of replicating the MySQL database (salesdb) to Amazon MSK utilizing Debezium, an open-source connector. The connector will monitor for any modifications to the database and seize any modifications to the tables.

With MSK Join, you may run absolutely managed Apache Kafka Join workloads on AWS. MSK Join provisions the required sources and units up the cluster. It constantly screens the well being and supply state of connectors, patches and manages the underlying {hardware}, and auto scales connectors to match modifications in throughput. In consequence, you may focus your sources on constructing purposes relatively than managing infrastructure.

MSK Join will make use of the customized DNS server within the VPC and it received’t be depending on Route 53.

Create an MSK cluster configuration

Full the next steps to create an MSK cluster:

  1. On the Amazon MSK console, select Cluster configurations underneath MSK clusters within the navigation pane.
  2. Select Create configuration.
  3. Identify the configuration mskc-tutorial-cluster-configuration.
  4. Underneath Configuration properties, take away every part and add the road auto.create.matters.allow=true.
  5. Select Create.

Create an MSK cluster and fasten the configuration

Within the subsequent step, we connect this configuration to a cluster. Full the next steps:

  1. On the Amazon MSK console, select Clusters underneath MSK clusters within the navigation pane.
  2. Select Create clusters and Customized create.
  3. For the cluster identify, enter mkc-tutorial-cluster.
  4. Underneath Common cluster properties, select Provisioned for the cluster sort and use the Apache Kafka default model 2.8.1.
  5. Use all of the default choices for the Brokers and Storage sections.
  6. Underneath Configurations, select Customized configuration.
  7. Choose mskc-tutorial-cluster-configuration with the suitable revision and select Subsequent.
  8. Underneath Networking, select the MSK VPC.
  9. Choose the Availability Zones relying upon your Area, corresponding to us-east1a, us-east1b, and us-east1c, and the respective non-public subnets MSK-Personal-1, MSK-Personal-2, and MSK-Personal-3 in case you are within the us-east-1 Area. Public entry to those brokers must be off.
  10. Copy the safety group ID from Chosen safety teams.
  11. Select Subsequent.
  12. Underneath Entry management strategies, choose IAM role-based authentication.
  13. Within the Encryption part, underneath Between shoppers and brokers, TLS encryption might be chosen by default.
  14. For Encrypt knowledge at relaxation, choose Use AWS managed key.
  15. Use the default choices for Monitoring and choose Primary monitoring.
  16. Choose Ship to Amazon CloudWatch Logs.
  17. Underneath Log group, select go to Amazon CloudWatch Logs console.
  18. Select Create log group.
  19. Enter a log group identify and select Create.
  20. Return to the Monitoring and tags web page and underneath Log teams, select Select log group
  21. Select Subsequent.
  22. Evaluation the configurations and select Create cluster. You’re redirected to the main points web page of the cluster.
  23. Underneath Safety teams utilized, notice the safety group ID to make use of in a later step.

Cluster creation can usually take 25–half-hour. Its standing modifications to Lively when it’s created efficiently.

Replace the /var/named/ zone file

Earlier than you create the MSK connector, replace the DNS server configurations with the MSK cluster particulars.

  1. To get the listing of bootstrap server DNS and respective IP addresses, navigate to the cluster and select View shopper info.
  2. Copy the bootstrap server info with IAM authentication sort.
  3. You’ll be able to determine the dealer IP addresses utilizing nslookup out of your native machine and it’ll present you the dealer native IP tackle. At the moment, your VPC factors to the newest DHCP choice set and your DNS server won’t be able to resolve these DNS names out of your VPC.
    nslookup <dealer 1 DNS identify>

Now you may log in to the DNS server and replace the data for various brokers and respective IP addresses within the /var/named/ file.

  1. Add the msk-access.pem file to BastionHostInstance out of your native machine:
    scp -i "< your pem file>" Your pem file ec2-user@<BastionHostInstance IP tackle>:/residence/ec2-user/

  2. Log in to the DNS server and open the /var/named/ file and replace the next traces with the right MSK dealer DNS names and respective IP addresses:
    <b-1.<clustername>.******.c6> IN A <Inside IP Handle - dealer 1>
    <b-2.<clustername>.******.c6> IN A <Inside IP Handle - dealer 2>
    <b-3.<clustername>.******.c6> IN A <Inside IP Handle - dealer 3>

Word that it’s worthwhile to present the dealer DNS as talked about earlier. Take away .kafka.<area id> from the dealer DNS identify.

  1. Restart the DNS service:
    sudo su
    service named restart

You need to see the next message:

Redirecting to /bin/systemctl restart named.service

Your customized DNS server is up and working now and you need to have the ability to resolve utilizing dealer DNS names utilizing the interior DNS server.

Replace the safety group for connectivity between the MySQL database and MSK Join

It’s necessary to have the suitable connectivity in place between MSK Join and the MySQL database. Full the next steps:

  1. On the Amazon MSK console, navigate to the MSK cluster and underneath Community settings, copy the safety group.
  2. On the Amazon EC2 console, select Safety teams within the navigation pane.
  3. Edit the safety group MySQL_SG and select Add rule.
  4. Add a rule with MySQL/Aurora as the kind and the MSK safety group because the inbound useful resource for its supply.
  5. Select Save guidelines.

Create the MSK connector

To create your MSK connector, full the next steps:

  1. On the Amazon MSK console, select Connectors underneath MSK Join within the navigation pane.
  2. Select Create connector.
  3. Choose Create customized plugin.
  4. Obtain the MySQL connector plugin for the newest steady launch from the Debezium website or obtain
  5. Add the MySQL connector zip file to the S3 bucket.
  6. Copy the URL for the file, corresponding to s3://<bucket identify>/
  7. Return to the Select customized plugin web page and enter the S3 file path for S3 URI.
  8. For Customized plugin identify, enter mysql-plugin.
  9. Select Subsequent.
  10. For Identify, enter mysql-connector.
  11. For Description, enter an outline of the connector.
  12. For Cluster sort, select MSK Cluster.
  13. Choose the present cluster from the listing (for this submit, mkc-tutorial-cluster).
  14. Specify the authentication sort as IAM.
  15. Use the next values for Connector configuration:
    database.historical past.producer.sasl.mechanism=AWS_MSK_IAM
    database.historical past.producer.sasl.jaas.config=software required;
    database.historical past.shopper.sasl.jaas.config=software required;
    database.historical past.kafka.matter=dbhistory.salesdb
    database.historical past.producer.sasl.shopper.callback.handler.class=software
    database.historical past.shopper.sasl.shopper.callback.handler.class=software
    database.historical past.shopper.sasl.mechanism=AWS_MSK_IAM

  16. Replace the next connector configuration:

  17. For Capability sort, select Provisioned.
  18. For MCU rely per employee, enter 1.
  19. For Variety of staff, enter 1.
  20. Choose Use the MSK default configuration.
  21. Within the Entry Permissions part, on the Select service function menu, select MSK-Join-PrivateDNS-MySQLConnector*, then select Subsequent.
  22. Within the Safety part, preserve the default settings.
  23. Within the Logs part, choose Ship to Amazon CloudWatch logs.
  24. Select go to Amazon CloudWatch Logs console.
  25. Underneath Logs within the navigation pane, select Log group.
  26. Select Create log group.
  27. Enter the log group identify, retention settings, and tags, then select Create.
  28. Return to the connector creation web page and select Browse log group.
  29. Select the AmazonMSKConnect log group, then select Subsequent.
  30. Evaluation the configurations and select Create connector.

Watch for the connector creation course of to finish (about 10–quarter-hour).

The MSK Join connector is now up and working. You’ll be able to log in to the MySQL database utilizing your person ID and make a few document modifications to the shopper desk document. MSK Join will have the ability to obtain CDC data and updates to the database might be out there within the MSK <Buyer> matter.

Devour messages from the MSK matter

To eat messages from the MSK matter, run the Kafka shopper on the MSK_Client EC2 occasion out there within the MSK VPC.

  1. SSH to the MSK_Client EC2 occasion. The MSK_Client occasion has the required Kafka shopper libraries, Amazon MSK IAM JAR file, file, and an occasion profile hooked up to it, together with the suitable IAM function utilizing the CloudFormation template.
  2. Add the MSKClientSG safety group because the supply for the MSK safety group with the next properties:
    • For Kind, select All Visitors.
    • For Supply, select Customized and MSK Safety Group.

    Now you’re able to eat knowledge.

  3. To listing the matters, run the next command:
    ./ --bootstrap-server <BootstrapServerString>

  4. To eat knowledge from the salesdb-server.salesdb.CUSTOMER matter, use the next command:
    ./ --bootstrap-server <BootstrapServerString> --consumer.config --topic salesdb-server.salesdb.CUSTOMER --from-beginning

Run the Kafka shopper in your EC2 machine and it is possible for you to to log messages just like the next:

Struct{after=Struct{CUST_ID=1998.0,NAME=Buyer Identify 1998,MKTSEGMENT=Market Section 3},supply=Struct{model=1.9.5.Ultimate,connector=mysql,identify=salesdb-server,ts_ms=1678099992174,snapshot=true,db=salesdb,desk=CUSTOMER,server_id=0,file=binlog.000001,pos=43298383,row=0},op=r,ts_ms=1678099992174}
Struct{after=Struct{CUST_ID=1999.0,NAME=Buyer Identify 1999,MKTSEGMENT=Market Section 7},supply=Struct{model=1.9.5.Ultimate,connector=mysql,identify=salesdb-server,ts_ms=1678099992174,snapshot=true,db=salesdb,desk=CUSTOMER,server_id=0,file=binlog.000001,pos=43298383,row=0},op=r,ts_ms=1678099992174}
Struct{after=Struct{CUST_ID=2000.0,NAME=Buyer Identify 2000,MKTSEGMENT=Market Section 9},supply=Struct{model=1.9.5.Ultimate,connector=mysql,identify=salesdb-server,ts_ms=1678099992174,snapshot=final,db=salesdb,desk=CUSTOMER,server_id=0,file=binlog.000001,pos=43298383,row=0},op=r,ts_ms=1678099992174}
Struct{earlier than=Struct{CUST_ID=2000.0,NAME=Buyer Identify 2000,MKTSEGMENT=Market Section 9},after=Struct{CUST_ID=2000.0,NAME=Buyer Identify 2000,MKTSEGMENT=Market Segment10},supply=Struct{model=1.9.5.Ultimate,connector=mysql,identify=salesdb-server,ts_ms=1678100372000,db=salesdb,desk=CUSTOMER,server_id=1,file=binlog.000001,pos=43298616,row=0,thread=67},op=u,ts_ms=1678100372612}

Whereas testing the applying, data with CUST_ID 1998, 1999, and 2000 have been up to date, and these data can be found within the logs.

Clear up

It’s all the time follow to wash up all of the sources created as a part of this submit to keep away from any further value. To wash up your sources, delete the MSK Cluster, MSK Join connection, EC2 situations, DNS server, bastion host, S3 bucket, VPC, subnets and CloudWatch logs.

Moreover, clear up all different AWS sources that you just created utilizing AWS CloudFormation. You’ll be able to delete these sources on the AWS CloudFormation console by deleting the stack.


On this submit, we mentioned the method of organising MSK Join utilizing a non-public DNS. This characteristic lets you configure connectors to reference public or non-public domains.

We’re capable of obtain the preliminary load and CDC data from a MySQL database hosted in a separate VPC and its DNS will not be accessible or resolvable externally. MSK Join was in a position to connect with the MySQL database and eat the data utilizing the MSK Join non-public DNS characteristic. The customized DHCP choice set was hooked up to the VPC, which ensured DNS decision was carried out utilizing the native DNS server as a substitute of Route 53.

With the MSK Join non-public DNS help characteristic, you may make your databases, knowledge warehouses, and methods like secret managers that work with your personal VPC inaccessible to the web and have the ability to overcome this limitation and comply along with your company safety posture.

To be taught extra and get began, check with non-public DNS for MSK join.

Concerning the writer

Amar is a Senior Options Architect at Amazon AWS within the UK. He works throughout energy, utilities, manufacturing and automotive prospects on strategic implementations, specializing in utilizing AWS Streaming and superior knowledge analytics options, to drive optimum enterprise outcomes.



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments