Mysql Cluster Community Edition 7.1

This week, I’ve set up mysql cluster on virtual machines to do some benchmarking and testing against performance.

For the basis I have implemented the steps outlined here.

I ran into a few snags however.  The first snag was that I was using virtual machines from rackspace cloud with 256MB of ram.  Surely I could run mysql with only 256MB of ram right?  In 7.1 they allowed disk access to the database so I should be fine, right?  – NOPE.


Mar 17 21:01:09 db2 kernel: [11925.786630] ndbd invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0
Mar 17 21:01:09 db2 kernel: [11925.786643] ndbd cpuset=/ mems_allowed=0
Mar 17 21:01:09 db2 kernel: [12298.051365] Pid: 1435, comm: ndbd Not tainted 2.6.35.4-rscloud #8


So I upgraded to a 512MB virtual machine – SAME THING. Apparently you need ATLEAST 1GB of ram. As once we upgraded to a 1GB virtual Machine we were able to fix this problem.

Two other issues that can cause this are SELINUX. To check run the below command:


cat /etc/selinux/config |grep ^SELINUX=


This should output: SELINUX=disabled.

Another reason can be in your cluster config.ini file – having the DataMemory and IndexMemory values too high can cause issues. The default settings that worked for me initially (just to start the service) were:

DataMemory=80M
IndexMemory=12M


Here’s something that cost me 2-3 hours of work: I initially had the DataMemory and IndexMemory set TOO high (extra 0) and it was trying to malloc 5GB of ram when only 2 GB of ram were in the machine. This was causing errors like:


2011-03-18 18:14:23 [ndbd] INFO -- NDB Cluster -- DB node 3
2011-03-18 18:14:23 [ndbd] INFO -- mysql-5.1.51 ndb-7.1.10 --
2011-03-18 18:14:23 [ndbd] INFO -- numa_set_interleave_mask(numa_all_nodes) : OK
2011-03-18 18:14:23 [ndbd] INFO -- Ndbd_mem_manager::init(1) min: 5268Mb initial: 5288Mb
ndbd_malloc_impl.cpp:561:grow(111030, 1152) 111030!=0 not using 36Mb - Unable to use due to bitmap pages missaligned!!
2011-03-18 18:14:23 [ndbd] ERROR -- ndbd_malloc_impl.cpp:565:grow(111030, 1152) not using 36Mb - Unable to use due to bitmap pages missaligned!!


I then changed the values in config.ini and re-ran the below thinking it would reload the config, right?:
ndb_mgmd -f /var/lib/mysql-cluster/config.ini

This DID NOT work. So then I tried –reload:
ndb_mgmd -f /var/lib/mysql-cluster/config.ini --reload

I’m smart, I read the man entry…..wtf – this didn’t work again. I’m getting sick of re-booting these machines (thank goodness for VM technology – NOC guys would have hated me right about now).

Seems you need to call it with the –initial flag AGAIN in order to flush the config.

Beginning with MySQL Cluster NDB 6.4.0, configuration data is cached internally rather than being read from the cluster
global configuration file each time the management server is started (see Section 17.3.2, âMySQL Cluster Configuration
Filesâ
and then to re-read the configuration data from the cluster configuration file and to build a new cache.

So I ran
ndb_mgmd -f /var/lib/mysql-cluster/config.ini --initial

And then started ndbd on the node machines and WOOO – it worked!

Hope this article saved somebody an hour or 3! Documentation and thorough articles on this stuff is hard to come by. Leave me a post in the comments field if this helped you.