Migrate from legacy MongoDB to new MongoDB based on Kubernetes
Page last updated:
We have a completely new MongoDB service (managed by Kubernetes Container Orchestration) in store for you. It comes with many improvements (MongoDB version 3.4.10 and high availability with replica set) and we recommend to migrate as soon as possible.
$ cf marketplace | grep mongodb mongodb-2 small*, medium*, large* MongoDB Replica Set v3.4.10 mongodbent small3rs*, medium3rs*, large3rs* MongoDB Enterprise HA v3.2.6
The different plans of our Kubernetes based MongoDB offering are at the moment:
$ cf marketplace -s mongodb-2 Getting service plan information for service mongodb-2 as admin... OK service plan description free or paid small Maximum 100 concurrent connections, 4 GB storage, 1 GB memory paid medium Maximum 200 concurrent connections, 8 GB storage, 2 GB memory paid large Maximum 400 concurrent connections, 16 GB storage, 4 GB memory paid
Note: The offer mongodbent uses several VMs and is intended for large deployments. It is not affected by this migration guide.
Important: Please make sure that you have the latest cf CLI installed before you follow this migration guide.
First, create a new MongoDB instance and bind it to your app using the old MongoDB service:
$ cf create-service mongodb-2 your-plan my-db-new $ cf bind-service my-awesome-app my-db-new
Replace your-plan with small, medium or large.
Next, stop your app:
$ cf stop my-awesome-app
As a next step, we’ll have to install the MongoDB package (which contains mongodump, mongo Shell and mongorestore) on our local machine.
On macOS, you can use Homebrew. For other systems, please refer to your package manager or the instructions in the MongoDB Manual.
# macOS only $ brew install mongodb
Next, we’ll use cf ssh to access our services. As the proxy, you can either use an existing running app (Don’t use an app which is bound to your MongoDB service. These need to remain stopped for data consistency reasons.) in the same space or just push a dummy app if you don’t have one. This app will only be used to SSH into so we get access to our service instances.
Important: Please make sure that the app you are using has been pushed or restarted within the last week since it needs to use the latest security groups to be able to connect to the new MongoDB service.
# Only needed if you don't have a running app in your space $ git clone https://github.com/swisscom/cf-default-app-staticfile.git $ cd cf-default-app-staticfile $ cf push proxy-app --no-route
Now, we’ll need to create service keys for our service instances to get access credentials for them. To get these, run the following commands:
# Old database
$ cf create-service-key my-db migration
$ cf service-key my-db migration
{ ... }
# New database
$ cf create-service-key my-db-new migration
$ cf service-key my-db-new migration
{ ... }
We’ll use these credentials in the following steps, so remember how to get them. Please replace the terms in <...> with the respective value from these service keys.
Next, create an SSH tunnel to your old DB through the proxy app (or any other app you intend to use).
$ cf ssh proxy-app -L 13000:<old-db-host>:<old-db-port>
Now that we’re connected, we can dump the old database using mongodump. Open a new terminal and run the following command:
$ mongodump --host localhost:13000 --authenticationDatabase <old-db-database> -u <old-db-username> -d <old-db-database> Enter password: <old-db-password>
Note: if there’s an issue connecting to the database, try restaging the app you’re using as a proxy (and then recreate the SSH tunnel):
$ cf restage proxy-app
The last line in the terminal output is something like 2017-10-18T14:11:53.340+0200 done dumping database-name.collection-name (1769759 documents).
# optional $ brew install tree
The tool mongodump creates the following tree in the current directory:
$ tree dump
dump
└── database-name
├── collection-name.bson
└── collection-name.metadata.json
1 directory, 2 files
Next, create an SSH tunnel to the new MongoDB service. You need to connect to the current primary because it’s the only writable node.
Open the terminal where you have the SSH connection open, press ctrl+d and run the following command:
$ cf ssh proxy-app -L 13000:<new-db-host>:<new-db-port>
where new-db-port is the first port from the connection string (example from cf service-key my-db-new migration output mongodb://{ ... }@kubernetes-service-node.service.consul:41374,kubernetes-service-node.service.consul:47464,kubernetes-service-node.service.consul:42759/632913f0?replicaSet=rs_632913f0-bea7-4124-8ec7-eb4cefc02a0a)
Now use the mongo client in a new terminal window to see which node is the current primary:
$ mongo --quiet -u <new-db-username> localhost:13000/<new-db-database> --eval "rs.status().members.find(r=>r.state===1).name" -p Enter password: <new-db-password>
Note: if there’s an issue connecting to the database, try restaging the app you’re using as a proxy (and then recreate the SSH tunnel):
$ cf restage proxy-app
Exit the previous cf ssh session and use cf ssh with the correct primary node port.
$ cf ssh proxy-app -L 13000:<new-db-host>:<new-db-port>
Now we’ll use the mongorestore CLI to restore the dump we created:
$ cd dump/<old-db-database>/ # don't forget the "." at the end of the command $ mongorestore --host localhost:13000 --authenticationDatabase <new-db-database> -u <new-db-username> --writeConcern 2 -d <new-db-database> . Enter password: <new-db-password>
You should see something like 2017-10-18T15:16:31.129+0200 finished restoring 632913f0.collection-name (1769759 documents) on your screen. Afterwards, the database server will build your indexes (may take some time).
The old database is a single node MongoDB. In the new service, you need to take write concern and read concern into account. We recommend using a write concern of 2 in order to ensure data consistency at all times.
The old MongoDB offer provides MongoDB 3.2 and some customers are even using MongoDB 3.0 docker images. Please check Release Notes for MongoDB 3.4 and use the new features where it makes sense for your app.
If you need any support, please ask on Stack Overflow #swisscomdev.
Now, we can unbind the old MongoDB service and rename the new database to use the real name:
$ cf unbind-service my-awesome-app my-db $ cf rename-service my-db my-db-old $ cf rename-service my-db-new my-db
Now it’s time to start your app again with the new MongoDB service:
$ cf start my-awesome-app
For some apps, you’ll also need to do a restage so they can pick up all environment variables properly:
# Optional $ cf restage my-awesome-app
Also, depending on how your app gets the credentials, you might have to adjust it accordingly. For example the key for the credentials in the VCAP_SERVICES environment variable changed from mongodb to mongodb-2.
As soon as everything is working properly, you can remove the service keys and the old database:
$ cf delete-service-key my-old migration $ cf delete-service-key my-db-old migration $ cf delete-service my-db-old
You have now migrated your app to the new MongoDB service. Congratulations! Please repeat for all apps using the old one.
View the source for this page in GitHub