MySQL performance with Kubernetes: bare metal vs K8 with docker vs K8 with container

Why  database performance assessment with Kubernetes

Now a days, Kubernetes is being used to manage databases like MySQL/Mariadb  for reasons like manageability etc.  However, the real question is whether we are aware of the performance overhead costs.  So, the objective of my assessment is to find out actual overhead with different runtime like docker or container.

Approach 

I used, mariadb's open-source framework TAF , to run sysbench's  point-select memory bound test ( data fits into buffer-pool)  against latest MySQL image , 9.7.0 . Sysbench's memory bound point-select is  CPU-intensive test which stress system the most.

sysbench's 8 tables x 5 million rows ~1 GB data which  fits into 2 GB BP.  I have kept config same for  all tests.

I have done testing on my laptop - 6 core, 8 GB RAM, AMD Ryzen 5 7535U with Radeon Graphics

Results and Observations:

 

Threads Mysql 9.7.0
(bare metal)		 Mysql 9.7.0
(k8 port-forward+docker)		 Mysql 9.7.0
(k8 Nodeport+docker)		 Mysql 9.7.0
(k8 LB+docker)		 Mysql 9.7.0
(k8 Nodeport+containerd)		 Base bare metal
(%drop in k8 portforward+docker)		 Base bare metal
(%drop in k8 Nodeport+docker)		 Base bare metal
(%drop in k8 LB+docker)		 Base bare metal
(%drop in k8 Nodeport+conainerd)
1 28036.3 3559.63 12280.62 12528.32 21464.4 -87.30 -56.20 -55.31 -23.44
2 52371.82 6099.63 23091.35 23259.13 39372.24 -88.35 -55.91 -55.59 -24.82
4 86868.2 9208.46 39016.07 38880.1 65442.19 -89.40 -55.09 -55.24 -24.66
8 119438.23 11740.33 52888.53 51029.16 95500.06 -90.17 -55.72 -57.28 -20.04

  

  •  As you can see, in the above graph that K8 port-forward with docker is giving worst performance w.r.t to bare metal and  K8 with nodeport/LB are relatively better but still showing drop around 55%
  • Without docker , K8 with containerd  seems to improving but it has overhead of around 25%

 

TAF framework's result screenshot

 

 

 

  • it gives the details of results , config and machine details and everything is reported in html

 

I monitored using dim_stat and here is the snippet of few important graphs  like  select/sec, sessions, cpu% and BP usage .

Interesting fact is that soft-irq is high in k8 with docker/containerd for Nodeport and LB and cpu is max  around 75-80% , except port-forward case.


 

  

   

Detailed monitoring data, I have kept in github: here 

 

Misc

 TAF command example:

#to setup db 

perl ./taf.pl --prop=/home/amrendra/projects/kubernet-mysql-mariadb/sysbench_lua.properties --action=init-start-db-exit   --verbose --db-software-install-dir=/home/amrendra/TAF/database_software_installs/mysql-9.7.0-linux-glibc2.28-x86_64 --ignore-running-db-process 

#to run test

  perl ./taf.pl --prop=/home/amrendra/projects/kubernet-mysql-mariadb/sysbench_lua.properties --action=run-tests   --verbose --db-software-install-dir=/home/amrendra/TAF/database_software_installs/mysql-9.7.0-linux-glibc2.28-x86_64 --ignore-running-db-process

 

github links with result

  • TAF graph: https://github.com/amroo76/mysql-k8-performance/blob/main/K8_POINT_SELECT_20260528_103253.chartplus_testinfo.html
  • TPS comparison: https://github.com/amroo76/mysql-k8-performance/blob/main/tps-graph.ods
  • Monitoring:  https://github.com/amroo76/mysql-k8-performance/blob/main/monitoring-report.pdf 

 (Calico seems to be working fine everytime rather than  Flannel)

setup of mysql in k8 with docker/container

refer github

References:

  • https://github.com/MariaDB/TAF 
  • https://docs.docker.com/engine/install/ubuntu/
  • https://docs.docker.com/engine/install/linux-postinstall/
  • https://medium.com/@midejoseph24/deploying-mysql-on-kubernetes-16758a42a746  
  • https://medium.com/@Adekola_Olawale/part-12-setting-up-a-local-kubernetes-cluster-with-minikube-56c98660189d#b5b1 
  • https://www.digitalocean.com/community/tutorials/bare-metal-kubernetes 

 

 

 

Comments

Post a Comment

Popular posts from this blog

MySQL- Crash Recovery Performance Assessment

Why we need to assess the performance of   crash recovery in MySQ and here is my explanation:  It’s not just WLs or bug fixes — any change in redo format or code changes around it, flushing behavior, or recovery concurrency can silently slow things down and if you have huge volume of data to recover than it could be painful. MySQL errorlog is the best place to look into details like whats happening during recovery and which phase of recovery is time consuming. Basically , I measured the recovery  timing  - from “Starting crash recovery” to “Apply batch completed , in MySQLD error log. Refer this about recovery in Innodb here: https://dev.mysql.com/doc/refman/8.4/en/innodb-recovery.html#innodb-crash-recovery Let me share some of my hack to measure the time taken for crash-recovery experiment. 1.  I used ` innodb_checkpoint_disabled ` which is strictly a testing mechanism. It forces maximum redo accumulation and doesn’t reflect real‑world recovery .   in...
Read more

Mariadb Performance : Openssl and wolfssl

Image
Why this SSL performance assessment of Mariadb ? My first objective was to assess new openssl 3.5 or 3.6 and see whether it's improving for Mariadb  or not and second , I could not find any comparative analysis of ssl for Mariabd in public space. Earlier in MySQL, I had done exhaustive assessment of  wolfssl and openssl's different version so thought of doing some against Mariadb which is another popular database and moving fast in db ranking. I used  memory bound test and sysbench's point-select without prepared statement to stress TLS layer the most. Another test , I picked is connect/disconnect which fires point-select query for each connection like PHP and this also stressess TLS layer the most. Key Findings In case of sysbench's point-select query like " SELECT c FROM sbtest1 WHERE id = 12345 ;" , wolfssl (5.8.4 default) is the best. openssl-1.1.1l/3.0.19/3.5.5/3.6.1 are less upto 9% wrt wolfssl. In case of sysbench's point-select query like "...
Read more

PGO in MySQL community 9.7

Image
  PGO ( Profile-guided optimization) in mysql community edition of 9.7.0 -------------------------------------------------------------------------------------------------------------------------- Straightforward ,  30% gain   we have seen in eneterprise edition of mysql   in read only sysbench test like point-select ( memory bound) .  I am curious to know whether similar gain , we see in pgo community build and what kind of test cases it use to train data? If pgo in community edition gives 30 percentage gain then it makes sense to use this feature by many other databases based on MySQL because it comes free without doing any extra tuning of mysqld config. As you can see,9.7.0 gives better performance wrt 9.6.0 and it's because of just pgo being put in. I referred INFO_BIN and 9.7 has fprofile compiler option whereas 9.6.0 doesn't have So, far I have seen release notes and I know, there is no other major changes would have contributed such gain. Its go...
Read more