Moving from django-partial-date#

This page moves a column written by django-partial-date to a PartialDateField, keeping every value. django-partial-date stores a timestamp and hides the precision in its seconds: 1990-01-01 00:00:00 is the year 1990, 1990-06-01 00:00:01 is June 1990 and 1990-06-14 00:00:02 is 14 June 1990. The package gives you one migration operation that rewrites those timestamps as 1990, 1990-06 and 1990-06-14, and one field that lets your existing migrations stop importing django-partial-date.

The operation converts a value only when it is exactly what django-partial-date writes. Any other value stops the migration before anything changes. You do the move in the order of the sections below: point your old migrations at the stand-in, change your models, convert, correct whatever the conversion refuses, and uninstall django-partial-date.

The examples come from the package’s own test app, moving, which is at tests/moving. Read tests.moving as the Python path of your own app. A code block that names a file under tests/moving is that file, word for word.

Install this package#

Install django-partial-dates into the environment your project runs in, before you change anything.

$ pip install django-partial-dates

The installation page says what the package sets up. Keep it installed after the move: your migrations refer to it.

Point existing migrations at the stand-in#

Use OldPartialDateField in the migrations that created or altered the column, in place of the field from django-partial-date. It describes the same column, so those migrations still create, alter and rebuild the table as they did, and the project can uninstall django-partial-date.

  1. In each migration that names the old field, change the import and the reference. This is the whole edit.

    import partial_date.fields

field = partial_date.fields.PartialDateField(blank=True, null=True)

    import partial_dates.conversion

field = partial_dates.conversion.OldPartialDateField(blank=True, null=True)

  2. Keep every option the migration passed. OldPartialDateField takes any option of Django’s Field, and no max_length.

The first migration of the test app creates two models, one with a nullable, indexed field that has its own db_column, and one with a required field:

from django.db import migrations, models

import partial_dates.conversion


class Migration(migrations.Migration):
    initial = True

    dependencies = []

    operations = [
        migrations.CreateModel(
            name="Sighting",
            fields=[
                (
                    "id",
                    models.BigAutoField(
                        auto_created=True,
                        primary_key=True,
                        serialize=False,
                        verbose_name="ID",
                    ),
                ),
                (
                    "seen",
                    partial_dates.conversion.OldPartialDateField(
                        blank=True,
                        db_column="seen_on",
                        db_index=True,
                        help_text="When it was seen.",
                        null=True,
                        verbose_name="seen",
                    ),
                ),
            ],
        ),
        migrations.CreateModel(
            name="Survey",
            fields=[
                (
                    "id",
                    models.BigAutoField(
                        auto_created=True,
                        primary_key=True,
                        serialize=False,
                        verbose_name="ID",
                    ),
                ),
                (
                    "taken",
                    partial_dates.conversion.OldPartialDateField(
                        help_text="When the survey was taken.", verbose_name="taken"
                    ),
                ),
            ],
            options={"abstract": False},
        ),
    ]

A migration that renames the field or alters one of its options changes in the same way, and keeps its other operations:

from django.db import migrations

import partial_dates.conversion


class Migration(migrations.Migration):
    dependencies = [("moving", "0001_initial")]

    operations = [
        migrations.RenameField(
            model_name="sighting", old_name="seen", new_name="observed"
        ),
        migrations.AlterField(
            model_name="sighting",
            name="observed",
            field=partial_dates.conversion.OldPartialDateField(
                blank=True,
                db_column="seen_on",
                db_index=True,
                help_text="When the animal was observed.",
                null=True,
                verbose_name="observed",
            ),
        ),
    ]

Use the stand-in in migrations only. A model always declares PartialDateField.

Rebase your own subclass#

If your project subclasses django-partial-date’s field, change both places that name it.

  1. In your migrations, replace the subclass with OldPartialDateField, as in the section above. The second model in 0001_initial.py above shows it.

  2. In your code, base the subclass on PartialDateField.

    from partial_date.fields import PartialDateField


class ProjectDateField(PartialDateField):
    """A project's own subclass of the date field."""

from partial_dates import PartialDateField


class ProjectDateField(PartialDateField):
    """A project's own subclass of the date field, as it reads after the move."""

  1. Check what the subclass overrides. Anything that depended on the old field storing a timestamp no longer applies, because the field now stores the ISO form.

Change the models#

Change each model to PartialDateField, or to your rebased subclass, with the options the field already has. Models that inherit the field from an abstract base change in the base, and each concrete model gets its own conversion in the next section.

from django.db import models

from partial_dates import PartialDateField
from tests.moving.fields import ProjectDateField


class Sighting(models.Model):
    observed = PartialDateField(
        null=True,
        blank=True,
        db_index=True,
        db_column="seen_on",
        verbose_name="observed",
        help_text="When the animal was observed.",
    )
    note = models.CharField(max_length=20, blank=True, db_default="")


class Dated(models.Model):
    taken = ProjectDateField(
        verbose_name="taken", help_text="When the survey was taken."
    )

    class Meta:
        abstract = True


class Survey(Dated):
    note = models.CharField(max_length=20, blank=True, db_default="")

Convert the columns#

Use ConvertPartialDateField to change a django-partial-date column into a PartialDateField in place. It takes the arguments of Django’s own AlterField.

  1. Make the migration. Django writes an AlterField for each change of field.

    $ python manage.py makemigrations moving

  2. In the migration Django wrote, change the name of each AlterField to ConvertPartialDateField, and import it. Nothing else changes. The test app’s conversion does it for one field of each kind:

from django.db import migrations

import partial_dates.fields
import tests.moving.fields
from partial_dates.conversion import ConvertPartialDateField


class Migration(migrations.Migration):
    dependencies = [("moving", "0003_add_note")]

    operations = [
        ConvertPartialDateField(
            model_name="sighting",
            name="observed",
            field=partial_dates.fields.PartialDateField(
                blank=True,
                db_column="seen_on",
                db_index=True,
                help_text="When the animal was observed.",
                null=True,
                verbose_name="observed",
            ),
        ),
        ConvertPartialDateField(
            model_name="survey",
            name="taken",
            field=tests.moving.fields.ProjectDateField(
                help_text="When the survey was taken.", verbose_name="taken"
            ),
        ),
    ]

  1. Apply the migration to a copy of your data first.

    $ python manage.py migrate moving

Every value becomes the ISO form of the partial date it recorded, and every empty value stays empty. The column then has the type, the check, the indexes and the constraints that a new PartialDateField column has, and makemigrations finds nothing more to do. Migrating backwards restores the timestamp column and the value each row held before.

Do not leave Django’s AlterField in place of the conversion. On a table that holds rows it fails with a database error and changes nothing, on PostgreSQL and on SQLite.

Three more things about the migration:

  • Leave it atomic, which is Django’s default. A refused value stops the migration before anything changes either way, but a failure after the conversion’s first step, such as a PostgreSQL view that depends on the column, leaves everything as it was only in an atomic migration.

  • Keep null as the old field had it. The conversion changes null in its last step, after the values are rewritten, so a change that the data refuses comes late. Make a change of null in its own alteration after the conversion.

  • The operation runs on PostgreSQL and SQLite and refuses any other database before it changes anything. A database router that excludes the model from a database excludes the conversion there too.

Correct a refused value#

When a value in the column is not one that django-partial-date writes, the conversion raises RefusedValuesError before it changes anything. Every row and the column are as they were, so you can correct the data and run the migration again.

The error has four attributes:

  • model is the label of the model, such as moving.Sighting.

  • field is the name of the field.

  • count is the number of refused values in the column.

  • rows holds the primary key and stored value of the first 20, lowest key first.

  1. Read count and rows in the error the migration printed.

    partial_dates.conversion.RefusedValuesError: moving.Sighting.observed holds 2 value(s)
django-partial-date does not write. The first are [(2, '1990-06-14 13:45:37'),
(7, '1990-06-14 00:00:00')].

  2. Decide what each row should hold. The table below says what each kind of value means.

  3. Correct each row, in a shell or in a data migration that runs before the conversion. Write the date at one of the three precisions: YYYY-01-01 00:00:00 for a year, YYYY-MM-01 00:00:01 for a month, YYYY-MM-DD 00:00:02 for a day. Set the column to NULL if the row should hold no date.

    UPDATE moving_sighting SET seen_on = '1990-06-14 00:00:02' WHERE id = 2;

  4. Apply the migration again. The count falls with each correction, and the migration completes when no value is refused.

Stored value

What it means

How to correct it

A time of day, seconds 3 to 59, or a fraction of a second, such as 1990-06-14 13:45:37

The column holds more than a date, because your project wrote its own timestamps into it.

Write the day it stands for at day precision, or NULL.

Year precision on a date other than 1 January, such as 1990-06-14 00:00:00

django-partial-date shows it as 1990, but the row also holds a month and a day.

Write 1990-01-01 00:00:00 to keep the year, or 1990-06-14 00:00:02 to keep the day.

Month precision on a day other than the first, such as 1990-06-14 00:00:01

django-partial-date shows it as 1990-06, but the row also holds a day.

Write 1990-06-01 00:00:01 to keep the month, or 1990-06-14 00:00:02 to keep the day.

A year outside 0001 to 9999, infinity, or a date before the common era (PostgreSQL)

The ISO form cannot hold it.

Write a date inside the range, or NULL.

Text that is not a timestamp, an impossible date, a number, empty text, or text with an offset (SQLite)

The column holds something other than a date.

Write one of the three precisions, or NULL.

The operation reads each value in the time zone of the database connection, which is the one django-partial-date used when it wrote it. If your project’s time zone settings changed after the rows were written, a value that no longer falls on 00:00:00, 00:00:01 or 00:00:02 there is refused instead of converted to a date one day out.

Uninstall django-partial-date#

Once every database you run has applied the conversion, and nothing in your project imports django-partial-date, remove it.

  1. Search your project for what still names it. The search finds partial_date and leaves out partial_dates, this package.

    $ grep -rnw partial_date .

  2. Remove django-partial-date from your requirements, then uninstall it.

    $ pip uninstall django-partial-date

  3. Check that your migrations load and that Django finds no change to make.

    $ python manage.py makemigrations --check --dry-run

Keep django-partial-dates in your requirements. Your migrations import partial_dates, and every new database is built from them.

Change the code that uses the old value class#

django-partial-date’s PartialDate and this package’s PartialDate are different classes. The table says what replaces each name of the old one.

Old class

What replaces it

PartialDate("1990-06")

PartialDate.fromisoformat("1990-06"). It reads YYYY, YYYY-MM and YYYY-MM-DD and nothing else, so "1990-6" raises ValueError, which the old class accepted. It raises ValueError, not ValidationError.

PartialDate(date, precision)

PartialDate(date.year), PartialDate(date.year, date.month) or PartialDate(date.year, date.month, date.day), according to the precision. A part you do not pass is None.

.date

.start, the first day of the period, or .end, the last. .year, .month and .day hold the recorded parts, and None for a part nobody recorded. A value cannot be assigned to. Build a new one.

.precision

.precision, which is a Precision member such as Precision.MONTH, not the integer 0, 1 or 2. It cannot be assigned to.

PartialDate.YEAR, PartialDate.MONTH, PartialDate.DAY

Precision.YEAR, Precision.MONTH and Precision.DAY, from partial_dates. They are not integers, so code that stored or computed with 0, 1 and 2 changes.

.precisionYear(), .precisionMonth(), .precisionDay()

.precision is Precision.YEAR, and likewise for month and day.

.format(year, month, day)

Nothing yet. Choose the format by .precision and apply it to .start, as in the example below.

str(value)

The same ISO form. A year below 1000 is padded, so str(PartialDate(990)) is "0990". repr(value) is no longer the ISO form, so code that showed repr uses str.

==

Equal when every part is equal, as before. 1990 is not 1990-01-01. Values are now hashable, so they work in sets and as dictionary keys.

<, <=, >, >=

Values sort in the order of their ISO forms, a year before its months and a month before its days. The old class compared date and precision separately, so two values could be neither greater nor smaller. Every pair is ordered now. Comparing with a date, a string or an integer raises TypeError.

This example runs each replacement:

from datetime import date

from partial_dates import PartialDate, Precision

year = PartialDate(1990)
month = PartialDate.fromisoformat("1990-06")
day = PartialDate(1990, 6, 14)

assert str(day) == "1990-06-14"
assert (day.year, day.month, day.day) == (1990, 6, 14)
assert day.start == date(1990, 6, 14) == day.end
assert month.precision is Precision.MONTH
assert year.month is None

if month.precision is Precision.MONTH:
    shown = month.start.strftime("%B %Y")
assert shown == "June 1990"

assert year < month < day
assert len({month, PartialDate(1990, 6)}) == 1

Queries change in two ways.

  • Ordering stays the same. Ordering by the field gives the order the old column gave, a year before the months of that year and a month before its days. The querying guide says where each database sorts a row holding no date.

  • Filters match what was recorded. On the old column the month and day lookups read the stored timestamp, so a date known only to the year matched month=1 and day=1. The lookups now match a part only on dates that recorded it. The querying guide says what each filter includes. A range takes a PartialDate or its ISO text as a bound. A date or a datetime raises ValueError, and lookups that only a timestamp has, such as hour, are gone. The start transform gives each date as a real date to build them on.

Check what the move does not cover#

Four things need your attention beyond the steps above.

  • A fixture with an unpadded year. django-partial-date dumps a year below 1000 without its leading zero, as 990-03. That is not an ISO form, and loaddata refuses it. Pad the year to four digits in the fixture, 0990-03. Fixtures from years 1000 onwards load unchanged, and so do empty values.

  • An earlier data migration that imports the old value class. Its import fails once django-partial-date is gone. Through the stand-in field a historical model gives the stored datetime, not a value class, and the seconds hold the precision: 0 for the year, 1 for the month and 2 for the day. Rewrite the migration to read and write that datetime.

  • The lock. The conversion takes an exclusive lock on the table until the migration ends, so nothing else reads or writes it meanwhile. On SQLite it rebuilds the table and locks the database for writing. The lock lasts as long as it takes to rewrite every row, so time the migration on your copy of the data.

  • Writes during the move. The check for refused values runs in Python before the conversion, and the conversion itself does not repeat it. Stop everything that writes the table from the start of the migration until it finishes, so that no row is written after the check. For the same reason, never run the output of sqlmigrate by itself. It holds the conversion’s SQL and no check, and run on a value the conversion would refuse it can write a date nobody recorded. Apply the migration with migrate.

A PostgreSQL view that depends on the column makes the database refuse the change, and the column stays as it was. Drop the view before the migration and create it again after.

Why a refused value stops everything#

Dropping or rounding a value that holds more than a partial date would lose something only you can judge. A row such as 1990-06-14 00:00:00 shows as 1990 in django-partial-date, but it also records a month and a day, and you may need them. So the conversion keeps nothing it has to guess: it checks every value first, and a single refused value leaves the whole table as it was.