Model Field - Django ORM Working - Part 2

The last post covered the structure of Django Model. This post covers how the model field works, what are the some important methods and functionality and properties of the field.

class Question(models.Model):
    question_text = models.CharField(max_length=200)
    pub_date = models.DateTimeField('date published')

Each class inherits from models.Model becomes a table inside the SQL database unless explicitly marked as abstract. The Question model becomes <app_name>_question table in the database. question_text and pub_date become columns in the table. The properties of the each field are declared by instantiating the respective class. Below is the method resolution order for CharField.

In [5]: models.CharField.mro()
Out[5]:
[django.db.models.fields.CharField,
 django.db.models.fields.Field,
 django.db.models.query_utils.RegisterLookupMixin,
 object]

CharField inherits Field and Field inherits RegisterLookUpMixin.

High-level role of Field class

The role of field class is to map type of the field to SQL database type.
Serialization - to convert the Python object into relevant SQL database value.
DeSerialization - to convert the SQL database value into Python object.
Check declared validations at the field level and built-in checks before serializing the data. For example, in a PositiveIntegerField the value should be greater than zero - built-in constraint.

Structure of Field Class

# Find out all the classes inheriting the Field

In [7]: models.Field.__subclasses__()
Out[7]:
[django.db.models.fields.BooleanField,
 django.db.models.fields.CharField,
 django.db.models.fields.DateField,
 django.db.models.fields.DecimalField,
 django.db.models.fields.DurationField,
 django.db.models.fields.FilePathField,
 django.db.models.fields.FloatField,
 django.db.models.fields.IntegerField,
 django.db.models.fields.IPAddressField,
 django.db.models.fields.GenericIPAddressField,
 django.db.models.fields.TextField,
 django.db.models.fields.TimeField,
 django.db.models.fields.BinaryField,
 django.db.models.fields.UUIDField,
 django.db.models.fields.json.JSONField,
 django.db.models.fields.files.FileField,
 django.db.models.fields.related.RelatedField,
 django.contrib.postgres.search.SearchVectorField,
 django.contrib.postgres.search.SearchQueryField,
 fernet_fields.fields.EncryptedField,
 enumchoicefield.fields.EnumChoiceField,
 django.contrib.postgres.fields.array.ArrayField,
 django.contrib.postgres.fields.hstore.HStoreField,
 django.contrib.postgres.fields.ranges.RangeField]

Here fernet_fields is a third-party library which implements the EncryptedField by inheriting the Field class. Also these are high level fields. For example, Django implements other high-level fields which inherit the above fields.

For example, EmailField inherits the CharField.

In [10]: models.CharField.__subclasses__()
Out[10]:
[django.db.models.fields.CommaSeparatedIntegerField,
 django.db.models.fields.EmailField,
 django.db.models.fields.SlugField,
 django.db.models.fields.URLField,
 django.contrib.postgres.fields.citext.CICharField,
 django_extensions.db.fields.RandomCharField,
 django_extensions.db.fields.ShortUUIDField]

Here is the Field class initializer signature

In [11]: models.Field?
Init signature:
models.Field(
    verbose_name=None,
    name=None,
    primary_key=False,
    max_length=None,
    unique=False,
    blank=False,
    null=False,
    db_index=False,
    rel=None,
    default=<class 'django.db.models.fields.NOT_PROVIDED'>,
    editable=True,
    serialize=True,
    unique_for_date=None,
    unique_for_month=None,
    unique_for_year=None,
    choices=None,
    help_text='',
    db_column=None,
    db_tablespace=None,
    auto_created=False,
    validators=(),
    error_messages=None,
)

The Field initializer contains 22 arguments. Most of the arguments are related to SQL database column properties and rest of the arguments are for Django admin and model forms.

For example, Django provides Admin interface to browse the database records and allows you to edit. blank parameter determines whether the field is required while filling up data in the admin interface and custom form. help_text field is used while display the form.

The most commonly used fields are max_length, unique, blank, null, db_index, validators, default, auto_created. null attribute is a boolean type when set to True, the allows null value while saving to the database. db_index=True created a B-Tree index on the column. default attribute stores the default value passed on to the database, when the value for the field is missing.

validators attribute contains list of validators passed on by the user and Django's internal validators. The function of the validator is to determine the value is valid or not. For example, in our question_text field declaration max_length is set to 200. When the field value is greater than 200, Django raises ValidationError. max_length attribute is useful only for text field and MaxLengthValidator will be missing in non-text fields.

In [29]: from django.core.exceptions import ValidationError

In [30]: def allow_odd_validator(value):
    ...:     if value % 2 == 0:
    ...:         raise ValidationError(f'{value} is not odd number')
    ...:

In [31]: int_field = models.IntegerField(validators=[allow_odd_validator])

In [32]: int_field.validators
Out[32]:
[<function __main__.allow_odd_validator(value)>,
 <django.core.validators.MinValueValidator at 0x1305fdac0>,
 <django.core.validators.MaxValueValidator at 0x1305fda30>]

In [33]: # let's look into question_text field validators

In [38]: question_text.validators
Out[38]: [<django.core.validators.MaxLengthValidator at 0x12e767fa0>]

As long as the validator function or custom class doesn't raise exception, the value is considered as valid.

The details of each field can found in the Django model field reference

Field Methods

In [41]: import inspect

In [44]: len(inspect.getmembers(models.Field, predicate=inspect.isfunction))
Out[44]: 59

In [45]: len(inspect.getmembers(models.Field, predicate=inspect.ismethod))
Out[45]: 6

The Field class consists of (along with inherited ones) 65 methods. Let's look at some of the important ones.

to_python

to_python method is responsible to convert the value passed on to the model during intialization. For example, to_python for IntegerField will convert the value to Python integer. The original value could be string or float. Every field will override to_python method. Here is an example of to_python method invocation on an IntegerField.

In [46]: int_field.to_python
Out[46]: <bound method IntegerField.to_python of <django.db.models.fields.IntegerField>>

In [47]: int_field.to_python('23')
Out[47]: 23

In [48]: int_field.to_python(23)
Out[48]: 23

In [49]: int_field.to_python(23.56)
Out[49]: 23

get_db_prep_value

get_db_prep_value method is responsible to convert Python value to SQL database specific value. Each field may have a different implementation depending on field type. For example, Postgres has a native UUID type, whereas in SQLite and MySQL Django uses varchar(32). Here is the implementation for get_db_prep_value from UUIDField.

def get_db_prep_value(self, value, connection, prepared=False):
    if value is None:
    return None
  if not isinstance(value, uuid.UUID):
    value = self.to_python(value)

  if connection.features.has_native_uuid_field:
    return value
  return value.hex

connection is a Database Connection or Wrapper object of underlying database. Below is an example output from a Postgres Connection and SQLite Connection for uuid field check.

In [50]: from django.db import connection
    ...:

In [51]: connection
Out[51]: <django.utils.connection.ConnectionProxy at 0x10e3c8970>

In [52]: connection.features
Out[52]: <django.db.backends.postgresql.features.DatabaseFeatures at 0x1236a6a00>

In [53]: connection.features.has_native_uuid_field
Out[53]: True

In [1]: from django.db import connection

In [2]: connection
Out[2]: <django.utils.connection.ConnectionProxy at 0x10fe3b4f0>

In [3]: connection.features
Out[3]: <django.db.backends.sqlite3.features.DatabaseFeatures at 0x110ba5d90>

In [4]: connection.features.has_native_uuid_field
Out[4]: False

One thing to note, Django uses psycopg2 driver for Postgres and it will take care of handling UUID specific to Postgres because UUID Python object needs to be converted to string or bytes before sending to the Postgres server.

Similar to get_db_prep_value, get_prep_value which converts Python value to query value.

formfield

Django supports ModelForm which is one to one mapping of HTML form to Django model. The Django admin uses ModelForm . The form consists of several fields. Each field in the form maps to field in the model. So Django can automatically construct the form with a list of validators from the model field.

Here is the implementation for the UUIDField.

def formfield(self, **kwargs):
    return super().formfield(**{
    'form_class': forms.UUIDField,
    **kwargs,
   })

When you create a custom database field, you need to create a custom form field to work with Django admin and pass it as an argument to super class method.

deconstruct

deconstruct method returns value for creating an exact copy of the field. The method returns a tuple with 4 values.

The first value is the name of the field passed during initialisation. The default value is None.
The import path of the field.
The list of positonal arguments passed during the field creation.
The dictionary of keyword arguments passed during the field creation.

In [62]: # Let's see the question_text deconstruct method return value

In [63]: question_text.deconstruct()
Out[63]: (None, 'django.db.models.CharField', 
[], {'max_length': 200})

In [65]: # let's create a new integer field with a name

In [66]: int_field = models.IntegerField(name='int_field', validators=[allow_odd_validator])

In [67]: int_field.deconstruct()
Out[67]:
('int_field',
 'django.db.models.IntegerField',
 [],
 {'validators': [<function __main__.allow_odd_validator(value)>]})

In [68]: models.IntegerField(**int_field.deconstruct()[-1])
Out[68]: <django.db.models.fields.IntegerField>

In [69]: int_2_field = models.IntegerField(default=2)

In [70]: int_2_field.deconstruct()
Out[70]: (None, 'django.db.models.IntegerField', 
[], {'default': 2})

Also when you implement a custom field, you can override the deconstruct method. Here is the deconstruct implementation for UUIDField.

def deconstruct(self):
    name, path, args, kwargs = super().deconstruct()
    del kwargs['max_length']
    return name, path, args, kwargs

init

__init__ method is a good place to override some of the default values. For example, UUIDField max_length should always be 32 irrespective of the value passed on. In the decimal field, max_digits can be modified during initialization.

Here is the UUIDField initializer method implementation.

def __init__(self, verbose_name=None, **kwargs):
    kwargs['max_length'] = 32
    super().__init__(verbose_name, **kwargs)

db_type

db_type method takes Django connection as an argument and returns the database specific implementation type for this field. The method takes connection as an argument. Here is the output of db_type for Postgres and SQLite.

In [72]: # Postgres

In [73]: uuid_field = models.UUIDField()

In [74]: uuid_field.db_type(connection)
Out[74]: 'uuid'

In [8]: # Sqlite

In [9]: uuid_field = models.UUIDField()

In [10]: uuid_field.rel_db_type(connection)
Out[10]: 'char(32)'

get_internal_type method returns internal Python type which is companion to the db_type method. In practice, Django fields type and database field mapping is maintained as class variable in DatabaseWrapper. You can find, Django fields and Postgres fields mapping in backends module. Below is the mapping taken from source code.

class DatabaseWrapper(BaseDatabaseWrapper):
    vendor = 'postgresql'
    display_name = 'PostgreSQL'
    # This dictionary maps Field objects to their associated PostgreSQL column
    # types, as strings. Column-type strings can contain format strings; they'll
    # be interpolated against the values of Field.__dict__ before being output.
    # If a column type is set to None, it won't be included in the output.
    data_types = {
        'AutoField': 'serial',
        'BigAutoField': 'bigserial',
        'BinaryField': 'bytea',
        'BooleanField': 'boolean',
        'CharField': 'varchar(%(max_length)s)',
        'DateField': 'date',
        'DateTimeField': 'timestamp with time zone',
        'DecimalField': 'numeric(%(max_digits)s, %(decimal_places)s)',
        'DurationField': 'interval',
        'FileField': 'varchar(%(max_length)s)',
        'FilePathField': 'varchar(%(max_length)s)',
        'FloatField': 'double precision',
        'IntegerField': 'integer',
        'BigIntegerField': 'bigint',
        'IPAddressField': 'inet',
        'GenericIPAddressField': 'inet',
        'JSONField': 'jsonb',
        'OneToOneField': 'integer',
        'PositiveBigIntegerField': 'bigint',
        'PositiveIntegerField': 'integer',
        'PositiveSmallIntegerField': 'smallint',
        'SlugField': 'varchar(%(max_length)s)',
        'SmallAutoField': 'smallserial',
        'SmallIntegerField': 'smallint',
        'TextField': 'text',
        'TimeField': 'time',
        'UUIDField': 'uuid',
    }

get_internal_type values are keys and values are Postgres field names.

I have skipped the implementation of the rest of the methods like __reduce__ and check . You can go through the source code of Django fields in GitHub and also you will find class variables and private methods usages.

Django documentation has an excellent page on how-to write custom model field.

Summary

models.Field is the root of all the model fields.
Field initializer takes configuration details like name, default, db_index, null for the database columns and blank, help_text for non-column features like Django model form and Django admin.
__init__ method in the child class can override the user passed value and(or) set custom default value.
validators attribute in the field contains the user-defined validators and default validators specific to the field.
Every field needs to implement a few methods to work with the specific databases. Some of the useful methods are to_python, get_db_prep_value, get_prep_value, deconstruct, formfield, db_type.
Django Connection object or wrapper contains details and features of the underlying the database.