mongoengine-migrate

Migrations for MongoEngine inspired by Django

View the Project on GitHub bdragon300/mongoengine-migrate

Mongoengine-migrate

Installing
Overview
Command-line interface
Migrations

Framework-agnostic schema migrations for Mongoengine ODM. Inspired by Django migrations system.

WARNING: this is an unstable version of software. Please backup your data before migrating

Features

All mongoengine field types are supported, including simple types, lists, dicts, references, GridFS, geo types, generic types.

Installing

Requirements:

You can install this package using pip:

$ pip install mongoengine_migrate

Overview

When you make changes in mongoengine documents declarations (remove a field, for instance), it should be reflected in the database (this field should be actually removed from records). This tool detects such changes, creates migration file and makes needed changes in db. If you worked with migration systems for SQL databases, you should get the idea.

How it works

Unlike SQL databases the MongoDB is schemaless database, therefore in order to track mongoengine schema changes we’re needed to keep it somewhere. Mongoengine-migrate keeps current schema and migrations tree in a separate collection. By default it is “mongoengine_migrate”.

MongoDB commands

Mongoengine_migrate tries to make changes by using the fastest way as possible. Usually it is MongoDB update commands or pipelines. But sometimes this not possible because of too old version of MongoDB server.

For this case each modification command has its “manual” counterpart. It updates records by iterating on them in python code and makes manual update. This variant could be slower especially for big collections. It will be used automatically if MongoDB version is lower than required to execute a certain command.

Common workflow

  1. After you made changes in documents schema, run mongoengine_migrate makemigrations. Your documents will be scanned and compared to the versions currently contained in your migration files, and a new migration file will be created if changes was detected.
  2. In order to apply the last migration you run mongoengine_migrate migrate.
  3. Once the migration is applied, commit the migration and the models change to your version control system as a single commit - that way, when other developers (or your production servers) check out the code, they’ll get both the changes to your documents schema and the accompanying migration at the same time.
  4. If you’ll need to rollback schema to the certain migration, run mongoengine_migrate migrate <migration_name>.

Example

Let’s assume that we already have the following Document declaration:

from mongoengine import Document, fields
    
class Book(Document):
    name = fields.StringField(default='?')
    year = fields.StringField(max_length=4)
    isbn = fields.StringField()

Then we made some changes:

from mongoengine import Document, fields

# Add Author Document
class Author(Document):
    name = fields.StringField(required=True)

class Book(Document):
    caption = fields.StringField(required=True, default='?')  # Make required and rename
    year = fields.IntField()  # Change type to IntField
    # Removed field isbn
    author = fields.ReferenceField(Author)  # Add field

Such changes should be reflected in database. The following command creates migration file (myproject.db is a python module with mongoengine document declarations):

$ mongoengine_migrate makemigrations -m myproject.db

New migration file will be created:

from mongoengine_migrate.actions import *

# Existing data processing policy
# Possible values are: strict, relaxed
policy = "strict"

# Names of migrations which the current one is dependent by
dependencies = [
    'previous_migration'
]

# Action chain
actions = [
    CreateDocument('Author', collection='author'),
    CreateField('Author', 'name', choices=None, db_field='name', default=None, max_length=None,
        min_length=None, null=False, primary_key=False, regex=None, required=False,
        sparse=False, type_key='StringField', unique=False, unique_with=None),
    RenameField('Book', 'name', new_name='caption'),
    AlterField('Book', 'caption', required=True, db_field='caption'),
    AlterField('Book', 'year', type_key='IntField', min_value=None, max_value=None),
    DropField('Book', 'isbn'),
    CreateField('Book', 'author', choices=None, db_field='author', dbref=False, default=None,
        target_doctype='Author', null=False, primary_key=False, required=False, sparse=False,
        type_key='ReferenceField', unique=False, unique_with=None),
]

Next, upgrade the database to the latest version:

$ mongoengine_migrate migrate

You can rollback changes by downgrading to the previous migration:

$ mongoengine_migrate migrate previous_migration

Actual db changes

During the running forward the migration created above the following changes will be made:

On backward direction the following changes will be made:

Command-line interface

$ mongoengine_migrate --help
Usage: mongoengine_migrate [OPTIONS] COMMAND [ARGS]...

Options:
  -u, --uri URI                  MongoDB connect URI  [default:
                                 mongodb://localhost/mydb]

  -d, --directory DIR            Directory with migrations  [default:
                                 ./migrations]

  -c, --collection COLLECTION    Collection where schema and state will be
                                 stored  [default: mongoengine_migrate]

  --mongo-version MONGO_VERSION  Manually set MongoDB server version. By
                                 default it's determined automatically, but
                                 this requires a permission for 'buildinfo'
                                 admin command

  --log-level LOG_LEVEL          Logging verbosity level  [default: INFO]
  --help                         Show this message and exit.

Commands:
  downgrade       Downgrade db to the given migration
  makemigrations  Generate migration file based on mongoengine model changes
  migrate         Migrate db to the given migration. By default is to the last
                  one

  upgrade         Upgrade db to the given migration

There are several commands exist. Each command has its own help available by running mongoengine_migrate <command> --help.

Dry run mode

downgrade, upgrade, migrate have “dry-run mode” when they just print commands which would be executed without actually applying\unapplying migration and making changes in database. Use --dry-run flag to run command in this mode.

Bear in mind that actual MongoDB commands could be slightly different from printed ones in this mode, because the tool sees unchanged database, but behavior of some commands could depend on db changes made by previous commands.

Migrate without changes

Use --schema-only flag to apply migration without making any changes in database. It could be suitable if you want to upgrade\downgrade database to this migration without making any changes in database.

This mode is equivalent to temporarily making all actions as “dummy”.

MongoDB version

Usually the version of MongoDB determines automatically. But this process requires right to run “buildinfo” command on server. If it not possible, you can specify version manually using --mongo-version argument.

Migrations

Migration is a file which contains instructions which have to be executed in order to apply this migration. These files are actually normal Python files with an agreed-upon object layout, written in a declarative style.

Dependency graph

Every migration (except for the first one) always depends on at least one another migration. Dependencies are listed in dependencies variable.

For example:

# File: migration_3.py

dependencies = [
    'migration_1',
    'migration_2'
]

This means that migration_3 will be applied only after both migration_1 and migration_2 has been applied before. And vice versa, in order to unapply migration_1, the migration_3 must be unapplied first.

Hereby all migrations are organized in one directed graph. When you creating a new migration file using makemigrations command, it will automatically fill dependencies with the last migrations in the graph.

Policy

Also a migration can set policy of changes handling. Available policies are following:

Actions

Every migration consists of instructions called “actions”. They are contained in actions variable.

For example:

from mongoengine_migrate.actions import *
import pymongo

actions = [
    AlterField('Book', 'year', type_key='IntField', min_value=None, max_value=None),
    CreateIndex('Book', 'caption_text', fields=[('caption', pymongo.TEXT)])
]

When you apply a migration, its actions are executed in order as they written in list: first the Book.year will be converted to integer, and then text index for Book.caption field will be created.

Order is reversed if you unapply a migration: first index will be dropped, and then Book.year will be converted back to string (as it was before migration).

An action represents one change, but can execute several commands. For example, more than one MongoDB command is required to execute to handle several parameters in AlterField action. For example, rename the field and convert its values to another type. Or updating embedded document schema.

Dummy action

Dummy action does not make changes in database. Such action is suitable when you want to deny it to make changes to database. If you will just remove action from the chain, then it will be appearing further in new migrations, because mongoengine-migrate won’t “see” the change of schema which this action should introduce to.

To make an action dummy, just add dummy_action=True to its parameters:

from mongoengine_migrate.actions import *

actions = [
# ...
    AlterField('Document1', 'field1', dummy_action=True, required=True, default=0),
# ...
]

Custom action

RunPython action is suitable when you want to call your own code in migration. You can have RunPython actions as many as you want.

For example:

from mongoengine_migrate.actions import *
from pymongo.database import Database
from pymongo.collection import Collection
from mongoengine_migrate.schema import Schema

def forward(db: Database, collection: Collection, left_schema: Schema):
    collection.update_one({'my_field': 1}, {'$set': {'your_field': 11}})

def backward(db: Database, collection: Collection, left_schema: Schema):
    collection.update_one({'my_field': 1}, {'$unset': {'your_field': ''}})

actions = [
# ...
# "forward_func" and "backward_func" are optional, but at least one of them must be set
    RunPython('Document1', forward_func=forward, backward_func=backward)
# ...
]

Callback functions parameters are:

  1. pymongo.Database object of current database
  2. pymongo.Collection object of collection of given mongoengine document. If document is embedded (its name starts with “~” symbol) then this parameter will be None.
  3. This parameter contains the Schema object modified by previous actions in chain.