Understanding Prefixed Auto-Incremented IDs in Django

Each model is typically assigned with an auto-incremented unique ID as the primary key. Whenever you define a model in Django, it will add this ID column/field to the model built-in if you don’t tell otherwise. Django and database thill increments with each new record automatically manage this ID. The AutoField in Django is a built-in field type that handles this behavior. For example:

from django.db import models

class Order(models.Model):
    id = models.AutoField(primary_key=True)

However, developers often require more than just a numerical ID. We might need a prefixed ID for better identification and sorting of records. For instance, adding a prefix such as ‘ORD’ to an order ID can make it more informative and easier to recognize.

Benefits of Using Prefixed IDs

Prefixed IDs in Django models offer a multitude of advantages over traditional numeric auto-incremented identifiers

#1 Readability

One significant benefit is the enhanced readability and context they provide. By including a prefix, IDs can immediately convey information about the type of object they represent, making them more intuitive for developers and users alike. For instance, an order with the ID ORD-1001 is easily identifiable as an order, unlike a nondescript number like 1001. This makes communicating the issues with users during the support ticket handling much more straightforward.

#2 Data Organization

Another key advantage is the improved organization and sorting of records. Prefixed IDs can help categorize data, particularly useful in systems with multiple entity types. This categorization can simplify data retrieval and manipulation and enhance the overall database management experience.

#3 Better Security

Moreover, using prefixed IDs can contribute to better security practices. They can obscure a system’s actual number of records, making it less noticeable to infer the dataset’s size or the records’ creation sequence. This can be a subtle yet effective deterrent against specific data enumeration attacks.

Exploring different methods for implementing prefixed ID fields

Implementing human-readable, queryable, prefixed, and automatically incremented ID-like fields requires a bit of customization but dramatically enhances the usability and readability of your data. To follow along with the provided examples, you first must set up a Django project.

Setting up the Django project

Setting up a Django project with PostgreSQL is a straightforward process. Here are the steps to follow:

1. Install PostgreSQL and create a new database for your project. Or use Docker to setup a PostgreSQL database
2. Update the DATABASES setting in your Django project’s settings.py file to use PostgreSQL as the database backend.
3. Install the psycopg2-binary package, which is the PostgreSQL adapter for Python.
4. Run the Django migrations to create the necessary tables in the database.

My Docker Compose setup for this tutorial:

# file: docker-compose.yml

version: "3"

# external services to connect to
services:

  postgres:
    image: postgres:12
    container_name: tutorial_postgres
    restart: always
    volumes:
      - tutorial-postgres:/var/lib/postgresql/data
    ports:
      - "5439:5432"
    env_file: .env
    healthcheck:
      test: ["CMD-SHELL", "sh -c 'pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}'"]
      interval: 5s
      timeout: 5s
      retries: 5

volumes:
  tutorial-postgres:

Django Settings:

# file: settings.py

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql_psycopg2",
        "NAME": os.environ.get("POSTGRES_DB"),
        "USER": os.environ.get("POSTGRES_USER"),
        "PASSWORD": os.environ.get("POSTGRES_PASSWORD"),
        "HOST": os.environ.get("POSTGRES_HOST"),
        "PORT": os.environ.get("POSTGRES_PORT"),
    }
}

Requirements:

# python:  3.12.2
# file: requirements.txt
asgiref==3.7.2
Django==5.0.2
psycopg2-binary==2.9.9
sqlparse==0.4.4

Starting models:

from django.db import models
from django.conf import settings

# Create your models here.

class Customer(models.Model):
    user = models.OneToOneField(
        settings.AUTH_USER_MODEL,
        on_delete=models.CASCADE
    )
    address = models.TextField()

    def __str__(self):
        return self.user.username

class Order(models.Model):
    customer = models.ForeignKey(Customer, on_delete=models.CASCADE)
    product_name = models.CharField(max_length=100)
    quantity = models.IntegerField()
    price = models.DecimalField(max_digits=10, decimal_places=2)
    order_date = models.DateField()

    def __str__(self):
        return f"{self.product_name} - {self.quantity}"

class Invoice(models.Model):
    order = models.OneToOneField(Order, on_delete=models.CASCADE)
    invoice_date = models.DateField()
    total_amount = models.DecimalField(max_digits=10, decimal_places=2)

    def __str__(self):
        return f"Invoice for Order: {self.order.id}"

Now lets explore different method to implement prefixed ID.

Method #1: Overriding save Method

Concept

The simplest way to start is by overriding the model’s save method. This method allows you to inject your logic for creating a custom ID before saving the model instance to the database.

Implementation

from django.db import models
from django.utils.translation import gettext_lazy as _

class Order(models.Model):
    order_id = models.CharField(max_length=10, unique=True, editable=False)
    # Other fields

    def save(self, *args, **kwargs):
        if not self.order_id:
            prefix = 'ORD-'
            last_order = Order.objects.all().values('order_id').order_by('id').last()
            if not last_order:
                new_id = 1
            else:
                number = int(last_order.get("order_id", 1).replace(prefix, '') or 1)
                new_id = number + 1
            self.order_id = prefix + str(new_id).zfill(5)
        super().save(*args, **kwargs)

Explanation

In this method, you check if the order_id is already set. If not, you generate a new one by finding the last order created, extracting its numeric part, incrementing it, and then concatenating it with the prefix and leading zeros to maintain the format.

Method #2: Django Signals

Concept

Django signals allow decoupling of applications by sending notifications when actions occur. A pre_save signal can be used to modify the instance before it’s saved without altering the model’s save method directly.

Implementation

# models.py

from django.db.models.signals import pre_save
from django.dispatch import receiver

...

class Invoice(models.Model):
    invoice_id = models.CharField(max_length=10, unique=True, editable=False)

    # other fields

...

@receiver(pre_save, sender=Invoice)
def set_invoice_id(sender, instance, *args, **kwargs):
    if not instance.invoice_id:
        prefix = 'INV'
        last_invoice = Invoice.objects.all().order_by('id').last()
        if not last_invoice:
            new_id = 1
        else:
            invoice_number = int(last_invoice.invoice_id.replace(prefix, ''))
            new_id = invoice_number + 1
        instance.invoice_id = prefix + str(new_id).zfill(4)

Explanation

This approach uses Django’s signal framework to listen for the pre_save event on the Invoice model. When an invoice is about to be saved and doesn’t have an invoice_id set, it calculates the new ID and sets it.

Method #3: Custom Model Field (Recommended)

Concept

Creating a custom model field allows you to encapsulate the logic for generating the custom ID, making your models cleaner and your custom ID logic reusable. The critical point in this method is “reusable.”

Implementation

from django.db import models

class PrefixedIDField(models.CharField):
    def __init__(self, *args, prefix='PRE', zfill=5, **kwargs):
        self.prefix = prefix
        self.zfill = zfill
        kwargs['max_length'] = kwargs.get('max_length', 10)  # default max_length is 10
        super().__init__(*args, **kwargs)

    def pre_save(self, model_instance, add):
        if add:
            last_id = model_instance.__class__.objects.all().order_by('-id').first()
            lastest_value = getattr(last_id, self.attname, None)
            if last_id:
                last_id = int(lastest_value.replace(self.prefix, '')) + 1
            else:
                last_id = 1
            value = f'{self.prefix}{str(last_id).zfill(self.zfill)}'
            setattr(model_instance, self.attname, value)
            return value
        return super().pre_save(model_instance, add)

class Customer(models.Model):
    customer_id = PrefixedIDField(prefix='CUST', unique=True, editable=False)

    # other fields...

Explanation

This custom field inherits from CharField and overrides the pre_save method to insert the logic for generating the custom ID. This method makes your models cleaner and the custom ID logic reusable across different models if needed.

Bonus Method: Model @property

Concept

The @property decorator in Python allows you to define a method in your class that can be accessed like an attribute. This feature can be used in Django models to create a custom formatted ID that combines a prefix with the existing auto-incremented id field of a model instance. This method does not change the actual ID in the database but provides a formatted string that can be used in the user interface, reports, or exports.

Implementation

from django.db import models

class Customer(models.Model):
    # Other fields as necessary

    @property
    def prefixed_id(self):
        """Generates a human-readable ID with a prefix."""
        return f"USR-{self.id:05d}"

In this example, the Customer model still uses Django’s default auto-incrementing id as its primary key. The @property named prefixed_id generates a string that combines a ‘USR-’ prefix with the id, formatted as a five-digit number with leading zeros.

Explanation

This method might seem advantages:

• Non-intrusive: It doesn’t require any changes to the existing database schema or the Django model’s primary key mechanism. This means it can easily be added to existing models without requiring data migration or schema modification.
• Performance: Because the underlying id field is still an integer, database indexing and lookup performance are not affected. The custom format is applied only when accessing the prefixed_id property, typically at the application level.
• Flexibility: The formatting logic is encapsulated within the model, making it easy to change the prefix or the formatting without affecting the rest of the application. If the requirements change, you only need to update the logic in one place.
• Readability: For user interfaces or external communications, displaying a more descriptive ID can be more user-friendly and professional. It makes IDs easier to read, communicate, and reference.

However, it is important to note that this method has some limitations and considerations:

• Data Integrity: The prefixed_id is not stored in the database as such. Therefore, when querying or filtering data, you must use the original id field. The prefixed_id is suitable for display purposes and should be used in situations where a more descriptive identifier benefits the user experience.
• Security: Exposing your ID field might not be a good idea in the long run. It is recommended to carefully consider the security implications of exposing internal identifiers to external systems or users.

Conclusion

By integrating custom ID fields with Django admin and forms, you can enhance the management and usability of your Django applications. This approach ensures that your custom IDs are correctly displayed and handled in the admin interface and custom forms, providing users with a seamless experience while maintaining the integrity and uniqueness of your IDs.

Not only does this improve the administrative capabilities of your Django application, but it also aligns with best practices for web application development. This way, you can ensure that your application remains robust, scalable, and easy to maintain.

Introduction to PostgreSQL views

What are the views in PostgreSQL?

Views in PostgreSQL are virtual tables based on a query’s result. They allow you to encapsulate complex queries into a single object that can be treated like a table. Views provide a way to simplify the complexity of querying data by abstracting away the underlying query logic.

Views can be used to:

• Simplify complex queries by breaking them down into smaller, more manageable pieces.
• Hide sensitive or confidential data by only exposing specific columns or rows.
• Provide a consistent interface for accessing data, even if the underlying table structure changes.

Views are handy when working with large and complex databases, as they allow you to organize and structure your data to make it easier to operate and understand.

Tip: When creating views, it’s essential to consider the performance implications. Views can introduce additional overhead, so optimizing them for the specific use case is necessary.

Advantages of using views

As a developer, one of the advantages of using views in PostgreSQL is the ability to abstract complex queries into a single, reusable object. This allows for cleaner, more maintainable code and improved performance.

Views also provide a way to simplify data access by presenting a logical representation of the underlying tables. This can be particularly useful when working with large and complex databases, as it allows us to focus on the specific data they need without understanding the underlying database structure. In my experience, these views are heavily used as a source for Microsoft Power BI or AWS QuickSight to visualize the data for dashboards.

Additionally, views can enhance security by limiting the data exposed to users. By granting access to views instead of tables, developers can control what data is visible and ensure that sensitive information is protected.

When working with views, it’s essential to remember that they are virtual tables and do not store any data. Instead, they provide a way to query and manipulate data from one or more tables in a structured and efficient manner.

Creating views in PostgreSQL

Creating views in PostgreSQL is a straightforward process. Views are virtual tables that are based on the result of a query. They allow us to encapsulate complex queries and reuse them like tables. To create a view, we use the CREATE VIEW statement followed by the view name and the query that defines the view.

Here’s an example of creating a view that shows the total sales for each product:

CREATE VIEW product_sales AS
SELECT product_id, SUM(quantity) AS total_sales
FROM sales
GROUP BY product_id;

This view can then be used in other queries or joined with other tables to retrieve the total sales for each product.

Using Django migrations with PostgreSQL views

Setting up Django project with PostgreSQL

Setting up a Django project with PostgreSQL is a straightforward process. Here are the steps to follow:

1. Install PostgreSQL and create a new database for your project. Or use Docker to setup a PostgreSQL database
2. Update the DATABASES setting in your Django project’s settings.py file to use PostgreSQL as the database backend.
3. Install the psycopg2-binary package, which is the PostgreSQL adapter for Python.
4. Run the Django migrations to create the necessary tables in the database.

My Docker Compose setup for this tutorial:

version: "3"

# external services to connect to
services:

  postgres:
    image: postgres:12
    container_name: tutorial_postgres
    restart: always
    volumes:
      - tutorial-postgres:/var/lib/postgresql/data
    ports:
      - "5439:5432"
    env_file: .env
    healthcheck:
      test: ["CMD-SHELL", "sh -c 'pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}'"]
      interval: 5s
      timeout: 5s
      retries: 5

volumes:
  tutorial-postgres:

Django Settings:

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql_psycopg2",
        "NAME": os.environ.get("POSTGRES_DB"),
        "USER": os.environ.get("POSTGRES_USER"),
        "PASSWORD": os.environ.get("POSTGRES_PASSWORD"),
        "HOST": os.environ.get("POSTGRES_HOST"),
        "PORT": os.environ.get("POSTGRES_PORT"),
    }
}

For demonstration purposes I’ve created orders app, and added models:

from django.db import models
from django.conf import settings

class Customer(models.Model):
    user = models.OneToOneField(
        settings.AUTH_USER_MODEL,
        on_delete=models.CASCADE
    )
    address = models.TextField()

    def __str__(self):
        return self.user.username

class Order(models.Model):
    customer = models.ForeignKey(Customer, on_delete=models.CASCADE)
    product_name = models.CharField(max_length=100)
    quantity = models.IntegerField()
    price = models.DecimalField(max_digits=10, decimal_places=2)
    order_date = models.DateField()

    def __str__(self):
        return f"{self.product_name} - {self.quantity}"

class Invoice(models.Model):
    order = models.OneToOneField(Order, on_delete=models.CASCADE)
    invoice_date = models.DateField()
    total_amount = models.DecimalField(max_digits=10, decimal_places=2)

    def __str__(self):
        return f"Invoice for Order: {self.order.id}"

Once you have completed these steps, your Django project will be set up to work with PostgreSQL. You can now start creating and using PostgreSQL views for reporting purposes.

Creating a migration for a PostgreSQL view

When creating a migration for a view in Django, it is important to understand the steps involved. First, you need to create a new migration file using the makemigrations command. This command will generate a new migration file that includes the necessary SQL statements to create the view. For detailed explanation check the Django’s official documentation.

# command
# python manage.py makemigrations --empty <yourappname> --name <migration_name>

# example
python manage.py makemigrations --empty orders --name raw_sql_view

>> Migrations for 'orders':
>>   orders/migrations/0002_raw_sql_view.py

Next, you can define the view using the migrations.RunSQL operation in the migration file. This operation allows you to execute raw SQL statements to create the view.

Here’s an example:

from django.db import migrations

class Migration(migrations.Migration):

    dependencies = [
        ('orders', '0001_initial'),
    ]

    operations = [
        migrations.RunSQL(
            'CREATE VIEW my_simple_view AS SELECT * FROM orders_invoice',
            'DROP VIEW my_simple_view'
        ),
    ]

Note that you need to specify both the SQL statement to create the view and the SQL statement to drop the view in case the migration needs to be rolled back.

Once the migration file is created, you can apply it using the migrate command. This will execute the SQL statements in the migration file and create the view in the database.

It is important to keep in mind that views are read-only, so you won’t be able to perform any write operations on them. If you need to update the view, you will need to create a new migration file that includes the necessary SQL statements to update the view.

Updating views with migrations

Updating views with migrations is a straightforward process in Django. Once a view has been created, any changes to the underlying tables or columns can be easily reflected in the view using migrations.

python manage.py makemigrations --empty orders --name update_raw_sql_view

>> Migrations for 'orders':
>>   orders/migrations/0003_update_raw_sql_view.py

To update a view with migrations, you can create a new migration file using the makemigrations command. In the migration file, you can use the RunSQL operation to execute the SQL statement that updates the view.

Here’s an example of how to update a view using migrations:

from django.db import migrations

class Migration(migrations.Migration):
    dependencies = [
        ("orders", "0002_raw_sql_view"),
    ]

    operations = [
        migrations.RunSQL(
            """
            CREATE OR REPLACE VIEW my_simple_view AS
            SELECT 
                inv.id,
                inv.invoice_date,
                inv.total_amount,
                inv.order_id,
                ord.product_name,
                ord.quantity,
                ord.price,
                ord.order_date,
                usr.first_name,
                usr.last_name
            FROM 
                orders_invoice inv
            INNER JOIN 
                orders_order ord ON inv.order_id = ord.id
            INNER JOIN 
                orders_customer cust ON ord.customer_id = cust.id
            INNER JOIN 
                auth_user usr ON cust.user_id = usr.id;
            """,
            "DROP VIEW my_simple_view",
        ),
    ]

As you can see when we want to have anything meaningful to show, SQL queries became more complex. But what if we could take advantage of Django ORM, to create PostgreSQL views?

Working with complex views in Django migrations

Using Django ORM in migrations

As a developer working with PostgreSQL views in Django migrations, one common scenario is the need to join multiple tables in a view. This allows us to combine data from different tables into a single virtual table that can be queried efficiently.

Create a new migration:

python manage.py makemigrations --empty orders --name django_orm_sql_view

>> Migrations for 'orders':
>>   orders/migrations/0004_django_orm_sql_view.py

Example of a migration with Django ORM:

from django.db import migrations

def generate_complex_qs():
    from orders.models import Invoice
    from django.db.models import F, Value
    from django.db.models.functions import Concat

    qs = Invoice.objects.annotate(
        customer_full_name=Concat(
            F("order__customer__user__first_name"),
            Value(" "),
            F("order__customer__user__last_name"),
        ),
    ).values(
        "id",
        "invoice_date",
        "total_amount",
        "customer_full_name",
        "order__customer__address",
        "order__product_name",
        "order__quantity",
        "order__price",
        "order__order_date",
    )
    return qs

def convert_qs_to_sql(qs):
    from django.db import connections

    cursor = connections["default"].cursor()
    sql_query = cursor.mogrify(*qs.query.sql_with_params()).decode("utf-8", "ignore")
    return sql_query

VIEW_NAME = "invoices_report"
VIEW_SQL = convert_qs_to_sql(generate_complex_qs())
CREATE_POSTGRES_VIEW_SQL = f"""
CREATE OR REPLACE VIEW {VIEW_NAME}
 AS
 {VIEW_SQL}
"""
REVERT_POSTGRES_VIEW_SQL = f"""
DROP VIEW IF EXISTS {VIEW_NAME};
"""

class Migration(migrations.Migration):
    dependencies = [
        ("orders", "0003_update_raw_sql_view"),
    ]

    operations = [migrations.RunSQL(CREATE_POSTGRES_VIEW_SQL, REVERT_POSTGRES_VIEW_SQL)]

Again to create the you should run migrate. Afterwards you can check if view created:

Using Django Models to access Postgres Views

Sometimes you’ll need to expose Postgres Views via API or use in your application. Thanks to Django Models you can do that easily, just defining managed=False in your models:

class InvoiceReport(models.Model):
    id = models.BigAutoField(primary_key=True)
    total_amount = models.DecimalField(max_digits=10, decimal_places=2)
    quantity = models.IntegerField()
    product_name = models.CharField(max_length=100)
    price = models.DecimalField(max_digits=10, decimal_places=2)
    order_date = models.DateField()
    invoice_date = models.DateField()
    customer_full_name = models.CharField()
    address = models.TextField()

    class Meta:
        managed = False
        db_table = "invoices_report"

    def __str__(self):
        return f"{self.customer_full_name} - {self.id}"

Django will require you to run makemigrations and migrate for your newly created InvoiceReport model, but under the hood it won’t try to re-create table or view:

./manage.py makemigrations

>> Migrations for 'orders':
>>   orders/migrations/0005_invoicereport.py
>>     - Create model InvoiceReport

./manage.py migrate       
>> Operations to perform:
>>   Apply all migrations: admin, auth, contenttypes, orders, sessions
>> Running migrations:
>>   Applying orders.0005_invoicereport... OK

Afterwards you can use InvoiceReport as usual Django Models:

./manage.py shell 
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from orders.models import * 
>>> 
>>> InvoiceReport.objects.all()
<QuerySet [<InvoiceReport: John Doe - 2>, <InvoiceReport: John Doe - 1>]>
>>> 
>>> for report in InvoiceReport.objects.all():
...     print(report.id, report.product_name, report.customer_full_name, report.total_amount)
... 
1 product #2 John Doe 10.10
2 product name #1 John Doe 15.98
>>>

Rolling back view migrations

Rolling back view migrations is a straightforward process in Django. When a view migration is rolled back, the corresponding view is dropped from the database.

To roll back a view migration, you can use the migrate command by passing the number or name of the previous migration. For detailed explanation check Django’s Documentation.

For example:

# python manage.py migrate <yourappname> <migration_name>

python manage.py migrate orders 0003_update_raw_sql_view

It’s important to note that rolling back a view migration will result in the loss of any data stored in the view. If you need to preserve the data in the view, you can create a new migration that recreates the view and then migrate to that new migration.

Tip: Before rolling back a view migration, make sure to backup any important data stored in the view to avoid data loss.

Optimizing PostgreSQL views for reporting purposes

Choosing the correct columns for the view

As a developer, when creating a PostgreSQL view for reporting purposes, it is important to carefully choose the columns that will be included in the view. Selecting the right columns can significantly impact the performance and usability of the view.

One approach is to include only the necessary columns that are relevant to the reporting requirements. This helps to minimize the amount of data retrieved from the database and improves query performance.

Additionally, consider including any calculated or derived columns that are commonly used in reporting. These columns can be pre-computed in the view, reducing the need for complex calculations in the reporting queries.

To summarize, when choosing the columns for a PostgreSQL view, focus on selecting the necessary columns and including any calculated or derived columns that are frequently used in reporting queries.

Here is a bulleted list of points to consider:

• Include only necessary columns
• Include calculated or derived columns
• Minimize data retrieval
• Optimize query performance
• Pre-compute commonly used calculations

Tip: By carefully selecting the columns for the view, you can improve the efficiency and effectiveness of your reporting queries.

Choosing Materialized Views over Views

When dealing with large datasets, views can simplify your code, but they don’t necessarily save you time during execution because their speed depends on the underlying query. This limitation becomes more evident with expensive queries and extensive datasets, presenting a drawback.

To address this concern for performance, a better alternative could be utilizing materialized views. These views allow you to store the query’s results on disk in a temporary table, acting as a cache. Consequently, querying the materialized view becomes much faster.

However, one drawback of materialized views is that they don’t automatically update when the data in the base tables changes. For instance, if a customer changes their address in the earlier example, we would see the updated information once we refresh the materialized view. This process involves rerunning the original query and caching the new results. An example demonstrating this situation will be presented in the next section.

Let’s create an example materializer view:

python manage.py makemigrations --empty orders --name django_orm_materialized_view

>> Migrations for 'orders':
>>   orders/migrations/0006_django_orm_materialized_view.py

Migration file:

from django.db import migrations

def generate_complex_qs():
    from orders.models import Invoice
    from django.db.models import F, Value
    from django.db.models.functions import Concat

    qs = Invoice.objects.annotate(
        customer_full_name=Concat(
            F("order__customer__user__first_name"),
            Value(" "),
            F("order__customer__user__last_name"),
        ),
    ).values(
        "id",
        "invoice_date",
        "total_amount",
        "customer_full_name",
        "order__customer__address",
        "order__product_name",
        "order__quantity",
        "order__price",
        "order__order_date",
    )
    return qs

def convert_qs_to_sql(qs):
    from django.db import connections

    cursor = connections["default"].cursor()
    sql_query = cursor.mogrify(*qs.query.sql_with_params()).decode("utf-8", "ignore")
    return sql_query

VIEW_NAME = "invoices_materialized_report"
VIEW_SQL = convert_qs_to_sql(generate_complex_qs())
CREATE_POSTGRES_VIEW_SQL = f"""
CREATE MATERIALIZED VIEW {VIEW_NAME}
 AS
 {VIEW_SQL};
CREATE UNIQUE INDEX ON {VIEW_NAME} (id);
"""
REVERT_POSTGRES_VIEW_SQL = f"""
DROP MATERIALIZED VIEW IF EXISTS {VIEW_NAME};
"""

class Migration(migrations.Migration):
    dependencies = [
        ("orders", "0005_invoicereport"),
    ]

    operations = [migrations.RunSQL(CREATE_POSTGRES_VIEW_SQL, REVERT_POSTGRES_VIEW_SQL)]

As you probably noticed, I created an INDEX on view. It’s required to be able to REFRESH the materialized views.

Result:

Refreshing materialized views

Refreshing materialized views is an important step in ensuring that the data in the view is up to date. Materialized views are a great way to improve query performance by precomputing and storing the results of a query. However, the data in a materialized view can become stale over time as the underlying data changes. To refresh a materialized view in PostgreSQL, you can use the REFRESH MATERIALIZED VIEW command.

Here is an example of how to refresh a materialized view:

REFRESH MATERIALIZED VIEW my_materialized_view;

This command will recompute the data in the materialized view based on the underlying tables and update the view with the latest data. It’s important to note that refreshing a materialized view can be an expensive operation, especially if the view contains a large amount of data or complex calculations. Therefore, it’s recommended to schedule regular refreshes based on the frequency of data changes and the performance requirements of your application.

To schedule regular refreshes of a materialized view, you can use the PostgreSQL REFRESH MATERIALIZED VIEW CONCURRENTLY command. This command allows you to refresh the view without locking the view, allowing concurrent reads and writes to the view while the refresh is in progress.

REFRESH MATERIALIZED VIEW CONCURRENTLY my_materialized_view;

By using the CONCURRENTLY option, you can minimize the impact on the performance of your application while keeping the materialized view up to date.

Let’s implement a refresh mechanism into our Django Model, so we can refresh in our app whenever we need a fresh data:

class InvoiceMaterializedReport(models.Model):
    id = models.BigAutoField(primary_key=True)
    total_amount = models.DecimalField(max_digits=10, decimal_places=2)
    quantity = models.IntegerField()
    product_name = models.CharField(max_length=100)
    price = models.DecimalField(max_digits=10, decimal_places=2)
    order_date = models.DateField()
    invoice_date = models.DateField()
    customer_full_name = models.CharField()
    address = models.TextField()

    class Meta:
        managed = False
        db_table = "invoices_materialized_report"

    def __str__(self):
        return f"{self.customer_full_name} - {self.id}"

    @classmethod
    def refresh_view(cl):
        with connection.cursor() as cursor:
            cursor.execute(
                f"REFRESH MATERIALIZED VIEW CONCURRENTLY {cl._meta.db_table}"
            )

Usage example:

./manage.py shell

>>> from orders.models import * 
>>> InvoiceMaterializedReport.refresh_view()
>>>

Handling view dependencies

When working with complex views in Django migrations, it is important to handle view dependencies properly. This ensures that the views are created and updated in the correct order, avoiding any errors or inconsistencies.

One way to handle view dependencies is by using the depends_on attribute in the migration file. This attribute specifies the views that the current view depends on, ensuring that the dependencies are resolved before creating or updating the view.

For example, let’s say we have two views: view1 and view2. If view2 depends on view1, we can specify this dependency in the migration file for view2 using the depends_on attribute:

from django.db import migrations

class Migration(migrations.Migration):

    dependencies = [
        ('app_name', '0001_initial'),
        ('app_name', '0002_view1_migration'),
    ]

    operations = [
        migrations.RunSQL(
            'CREATE VIEW view2 AS SELECT * FROM view1',
            'DROP VIEW view2'
        ),
    ]

By specifying the dependency on view1 in the dependencies list, Django ensures that view1 is created or updated before creating or updating view2.

Handling view dependencies correctly is crucial for maintaining the integrity and consistency of the database schema.

Conclusion

In conclusion, PostgreSQL views provide a powerful tool for creating custom reports in Django applications. By leveraging the capabilities of Django migrations, developers can easily define and manage views that can be used for reporting purposes. Views allow for efficient data retrieval and can be customized to meet specific reporting requirements. With the ability to perform complex queries and join multiple tables, views offer a flexible solution for generating meaningful insights from the database. PostgreSQL views are a valuable addition to any Django project, enabling developers to create comprehensive reports and analyze data in a structured and efficient manner.

Key Takeaways

• PostgreSQL views are virtual tables that can be used for reporting purposes.
• You can create and update views in your PostgreSQL database using Django migrations.
• Optimizing views involves choosing the correct columns, filtering and aggregating data, and using indexes for improved performance.
• Complex views can be created by joining multiple tables and using subqueries.

The client had a request: Is it possible to determine how energetically people are dancing?

The idea behind it was to measure people’s energy levels. As you may already guess, it was an ad campaign for one of the energy drink brands.

The campaign wasn’t designed to be an online campaign. The plan was that people will be dancing on the stage that was built for the ad campaign, and our software should be measuring their energy level during the dance in real-time.

But how?

Building

I’ve tried to design a system where high-speed internet is not an essential part of the software. If once we were able to open a page and javascript was loaded, it meant we’ll be able to operate as intended. Reducing the connection to the server meant most of the heavy lifting had to be done in the front end by Javascript. (after technical details, I’ll come back to this method and explain how it saved our operation big time)

This is how I designed the system:

The goal was every dancer should see his result in real-time on the big screen, to know how well he/she’s doing. In this way, the person performing would know if he/she should speed up or can slow down a bit if he/she is out of breath.

To achieve this, we had to show a progress bar that changes according to the dancer’s performance. And at the end of the performance, we had to send a recorded act of dance with the progress bar’s fluctuations to the server to store.

And last but not least, we had to be able to control the software remotely, because we shouldn’t interact with the notebook that mirrors the big screen.

Once requirements were established, it was time to build the actual system.

To determine and rate the energy level of the dancer, I decided to use motion detection technology. In other words, if there are too many movements in the frame, it’s probably an energetic dance. And to detect the motion using Javascript meant I had to compare every frame the camera sent me to see if there were enough movements and, if so, check how many pixels changed to rate the performance.

To rate the performance, I’ve set the minimum threshold to check if the amount of pixel changes were greater than the threshold, which meant the person on camera dancing energetically enough, and added 1% to his progress if pixel changes were below the threshold I penalized dancer by reducing his progress by 1%. This meant if the dancer decided to stop for a while to take a breath, his progress would keep declining. It would continue until the dancer hit 100%, which meant he won.

Suppose you want to dig a little deeper to understand the technical background of motion detection; I strongly recommend you to check out the articles I mentioned at the end of this note. Once you’ve understood how things work under the hood, you won’t feel like you’re shooting in the dark when you’re using a library. For example, I’ve used “diff-cam-engine” for motion detection.

When I was done with the game mechanism of the app, now I had to work on the part where I had to record the performance video, merge it with the progress bar animation, and add the music to the background, which played when the dancer performed.

To do it, I’ve decided to use hidden canvas, where I drew a video and the animation of the progress bar side by side in real-time. When the performer finished the act, I added the music to the background using WebRTC and sent the final product to the server. This was the only time I’d sent a huge chunk of data to the backend. (I’ve listed tutorials and code examples of the WebRTC at the end of this note)

Example of the canvas recording:

After dealing with the video editing part of the app, it was time to handle the remote control part of the system. If you remember, one of the requirements was that we shouldn’t interact with the monitor screen in any way. To achieve seamless integration with the backend, sockets came to the rescue. The front end of the app listened to the socket to receive the new player’s data and start and stop the command of the game. These pieces of information were sent from the app’s control panel, which listened to the socket to receive info about the process that happened on the app’s front end.

When it comes to the backend, Python is my go-to language. Because of Django’s solid structure and built-in features, I’ve decided to use Django at the backend this time as well. And handling the socket connections was fairly easy, too, because of Django channels. A simple Django setup inside the docker container did the trick. Here’s the setup example that I’ve used.

Conclusion

As you may already know, when you’re in the field, always something goes wrong. For example, in our situation, on the 2nd day of the event, the internet connection was so poor that uploading recorded videos to the server was a headache. And luckily, because of the system’s architecture and most of the processes being handled by javascript on the front side of the app, it wasn’t a big deal. We just turned off the video recording and uploading them to the server from the admin panel, and we were good to go. The app’s usage didn’t interrupt, and everybody was happy.

I was already familiar with video editing with FFmpeg at the backend and recording webcams with the help of ActionScript and Nginx RTMP. Still, the result of working on videos with Javascript was surprisingly positive too. And I’m looking forward to using it in future projects as well.

In conclusion, the project was completed successfully.

Below is a list of the articles and libraries that I looked into before starting the project may be helpful in your situation as well:

Libraries:

• https://github.com/lonekorean/diff-cam-engine
• https://github.com/beije/motion-detection-in-javascript
• https://github.com/ReallyGood/js-motion-detection
• https://trackingjs.com/examples/face_camera.html
• https://github.com/hdmchl/gest.js
• https://github.com/maniart/diffyjs
• https://github.com/collab-project/videojs-record
• https://recordrtc.org/
• https://github.com/spite/ccapture.js/
• https://github.com/Stanko/html-canvas-video-player

Articles:

• https://hackernoon.com/motion-detection-in-javascript-2614adea9325
• https://codersblock.com/blog/motion-detection-with-javascript/
• https://benjaminhorn.io/code/motion-detection-with-javascript-and-a-web-camera/
• https://www.adobe.com/devnet/archive/html5/articles/javascript-motion-detection.html
• https://hasgeek.tv/jsfoo/2013-1/681-using-camera-motion-detection-in-js-for-gestures-based-interaction
• https://webrtchacks.com/baby-motion-detector/

Back in 2013, I was spending an unhealthy amount of time on Twitter. And at that time, I was really into the caricatures. So instead of checking every site I know for any new caricatures, I thought wouldn’t be great if there were a Twitter account that “tweets” new caricatures every day. And actually, there were such accounts. The problem was that those accounts weren’t relevant enough and shared the same caricatures over and over… And most of them were running actively only for a couple of months. Because after a couple of months, the owner of accounts usually got bored and didn’t want to continue.

So to have an account without an owner that gets bored quickly, I thought we should delegate this task to the bots.

Building

Because it was a weekend project, I didn’t want it to cost an arm and leg. I knew that if I wanted the bot to be a long-running show, it should require a minimum of possible time and resources from me.

I was already familiar with Google Apps Script and had used it several times. So I thought if I could build the bot with Google Apps Script (GAS), it meant I wouldn’t be paying for servers, and as a bonus, I would get to play with the Javascript at the backend.

Obviously, it has been a long time since I started building it, and I can not recall all the process details. But I remember that there was built-in support for OAuth in GAS back then, so it was relatively easier to build a bot. And there was another service called ScriptDB in GAS. It allowed scripts to have databases and store data in them. For example, I’ve used ScriptDB to keep track of the caricatures to check if the bot has tweeted the picture before.

But both of those services were deprecated; I had to replace those parts along the way to keep the bot functioning. So when ScriptDB was deprecated, I started using Parse.com. Do you remember old, good Parse.com? The one that Facebook bought and killed after a while.

After the fate of Parse.com, I gave up on third-party databases, and since then, I’ve been using Google Sheets as a database for the bot.

And nowadays, you have to use libraries to authenticate with external services like Twitter:

• https://github.com/gsuitedevs/apps-script-oauth1
• https://github.com/gsuitedevs/apps-script-oauth2

Since the bot’s launch, it has seen a couple of iterations. Right now, it’s on 3rd version.

After the latest update, it now has a little web dashboard where you can manually post, check the time of scheduled events, or check the social channels’ connection status.

This is how it works:

You can set “triggers” on the Google Apps Script file. For example, I’ve set up a trigger that runs every hour and executes the GAS function that checks websites to see if any new caricatures are posted. When it finds a picture, it first runs it through the database (Google Sheets) to check if it’s a new picture. If so, then posts it to the social channels and write back to the database for future reference.

As you can see from the picture above, after figuring out how to integrate with Twitter, I’ve also incorporated a bot with Facebook and Tumblr. Tweeting or posting text was easy, but posting images with GAS took some time to figure out. So here’s the code snippet to save you time:

function getTwitterService() {
  // Check https://github.com/gsuitedevs/apps-script-oauth1#usage
  // for the docs
  return OAuth1.createService('twitter')
      // Set the endpoint URLs.
      .setAccessTokenUrl('https://api.twitter.com/oauth/access_token')
      .setRequestTokenUrl('https://api.twitter.com/oauth/request_token')
      .setAuthorizationUrl('https://api.twitter.com/oauth/authorize')

      // Set the consumer key and secret.
      .setConsumerKey(TWITTER.CONSUMER_KEY)
      .setConsumerSecret(TWITTER.CONSUMER_SECRET)

      // Set the name of the callback function in the script referenced
      // above that should be invoked to complete the OAuth flow.
      .setCallbackFunction('authCallback')

      // Set the property store where authorized tokens should be persisted.
      .setPropertyStore(PropertiesService.getUserProperties());
}

function tweetPicFromURL(url) {
  try {
    var boundary = "cuthere";
    var picture = UrlFetchApp.fetch(url).getBlob().setContentTypeFromExtension();
    var status = "Test status";

    var requestBody = Utilities.newBlob("--"+boundary+"\r\n"+
                                        "Content-Disposition: form-data; name=\"status\"\r\n\r\n"+status+"\r\n"+
                                        "--"+boundary+"\r\n"+
                                        "Content-Disposition: form-data; name=\"media[]\"; filename=\""+picture.getName()+"\"\r\n"+
                                        "Content-Type: "+picture.getContentType()+"\r\n\r\n").getBytes();
    requestBody = requestBody.concat(picture.getBytes());
    requestBody = requestBody.concat(Utilities.newBlob("\r\n--"+boundary+"--\r\n").getBytes());

    var options =
        {
          method: "post",
          contentType: "multipart/form-data; boundary="+boundary,
          payload: requestBody
        };

    // twitter stuff
    var api = 'https://api.twitter.com/1.1/statuses/update_with_media.json';
    var twitterService = getTwitterService();
    var response = twitterService.fetch(api, options);

    Logger.log(response);
    return response;
  }
  catch(e){Logger.log(e)}
}

Github Gist of the code snippet: https://gist.github.com/msadig/30dccada8104adc823336eaadce1cc6c

Conclusion

Honestly, up until this year’s (2018) Google I/O, I didn’t realize why this project appeals to me so much. But, after watching one of the presentations on Google I/O ‘18, I realized that I like this architecture and project because it’s serverless. And the fact that I was using serverless before even it was a “thing” made me love this project even more now.

So after 5 years and 11.3K caricatures that posted, I would like to think about this experiment as a successful one.

Here’re the links to check out for working bots:

• https://twitter.com/karikARTur
• http://fb.com/karikartur
• http://karikartur.tumblr.com/

Links:

• https://gsuite-developers.googleblog.com/2014/05/deprecating-scriptdb-and-domain-service.html
• https://developers.google.com/apps-script/migration/oauth-config
• https://techcrunch.com/2013/04/25/facebook-parse/
• https://ctrlq.org/code/19408-create-bot

When I say the cost-effective way of hosting a website, I don’t mean only the money side of hosting. It doesn’t make sense to me if I have to put a lot of time and effort into maintaining the site, even if I can host it for free. So when I was thinking about building a website to put my notes in, I knew I had to take into account a couple of things.

Serverless

Maintenance of the website shouldn’t be my abundance. From my past experiences, I knew that it’s a pain in the but to maintain a blog to keep its plugins and versions up to date. There will be security patches and updates; you must do those things almost instantly; otherwise, your resource will be vulnerable. It was obvious that I should take a “serverless” approach.

So WordPress.com was the logical place to start. I would be responsible for neither maintaining server support nor updates of the CMS. And because I was already familiar with WordPress, there wouldn’t be any learning curve either.

Customization

But when it comes to customizing the website, wordpress.com give little to none control over the blog. I would have some access to customize the blog if I’d choose a $8/month plan. But this option didn’t appeal to me too much. Not because it’s too expensive, but mainly because to be able to customize the site only from the WordPress Admin, and using just CSS, seemed bit limiting to me.

Although I wanted something minimalist, I still wanted to have total control over the website.

Stack

When it came to choosing the stack, again, I wanted it not to be time-consuming. Which meant I wanted to build with the stack that I was already familiar with before. I already knew about the static website concept but never tried it. A few times, I’ve read about Jekyll and how to use it, but it never appealed to me. The main reason was it didn’t feel intuitive to me to commit the blog post to the git. When I run these little experiments, I always try to think if it has real-world application. Meaning I never believed that I was gonna be able to find a blogger or content manager that easily, which going to adapt to the process of writing and formatting posts with the markdown and push it to the git as a commit.

Then one day, I stumbled upon Gatsby. The main attraction point for me was to have that many data sources. And, of course, React. That meant I could easily build a website using JAMStack with multitier architecture.

So architecture, I was thinking, was something like this:

• Bitbucket.org will serve as a git to store the project’s source code.
• WordPress.com will serve as an API for the website.
• Netlify.com will handle the build process and will serve as hosting as well.
• Cloudflare.com will handle caching and DNS management. And as a bonus website is going to get a free SSL
• And all of these are going to cost $0

So I started to build

This architecture allows me to keep the development and publishing environments separate. The editor or publisher works in the WordPress admin, and he/she does not have to commit or push the content changes. And with this structure, it’s possible to update the content even from a smartphone.

As you can see from the visual above, Gatsby is one of the most crucial parts of architecture. Gatsby is the glue that keeps together the whole scheme. Because of it, I can easily get the data from WordPress.com API and build customizable views for posts and pages. And because Gatsby uses GraphQL as a query language, it’s even more fun to work with it.

Gatsby’s official site covers all the steps well enough to know how to start and where to look. So I’m not going to copy and paste those things here. Instead, I encourage you to go and check the docs yourself: https://www.gatsbyjs.org/docs/

And because of I guilty of doing so, I highly recommend checking the community starters before starting the reinventing the wheel: https://www.gatsbyjs.org/docs/gatsby-starters/

But because tutorials usually show how to integrate with self-hosted WordPress sites, I’m gonna put a code piece below to save a couple of seconds of your life.

module.exports = {
  ...
  plugins: [
    ...,
    {
      resolve: `gatsby-source-wordpress`,
      options: {
        // your WordPress source
        baseUrl: `your.wordpress.com`,
        protocol: `https`,
        // is it hosted on wordpress.com, or self-hosted?
        hostingWPCOM: true,
        // does your site use the Advanced Custom Fields Plugin?
        useACF: false,
        auth: {
          // You'll get Client Secret and Client ID from https://developer.wordpress.com/apps/
          wpcom_app_clientSecret: 'client_secret_goes_here',
          wpcom_app_clientId: 'client_id_goes_here',
          // Your WordPress.com user credentials
          wpcom_user: 'email@example.com',
          wpcom_pass: 'veryverystrongpassword',
        },
        verboseOutput: true,
      }
    },
    ...,
  ]
}

The rest is pretty straightforward. You connect your Bitbucket or Github account with your Netlify.com account and whenever you push a new code, Netlify builds it and hosts it for you: https://www.netlify.com/docs/continuous-deployment/

Now, it’s time to set up a webhook to notify Netlify.com when we publish new content. You can do this from your WP Admin dashboard. You can get to this dashboard by adding /wp-admin to the end of your site’s URL (e.g., your.wordpress.com/wp-admin). You can read detailed instructions on how to set up webhook here: https://en.support.wordpress.com/webhooks/

And here you can learn how to get the URL for incoming hooks: https://www.netlify.com/docs/webhooks/

Now we have set up the continuous deployment process, it’s time to automate one more thing. As you can see from the scheme above, we rely on Cloudflare for our CDN. This means Cloudflare will cache static files to decrease the latency of user requests. And because our site was built just with static files, Cloudflare will cache our whole site. And we have to clear that cache somehow after every successful deployment.

Once again, Netlify hooks come to the rescue. This time outgoing webhook. Set up a webhook that visits the Cloudflare API:

https://www.cloudflare.com/api_json.html?a=fpurge_ts&tkn=<TOKEN>&email=<ACCOUNT_EMAIL>&z=<DOMAIN2PURGE.COM>&v=1

Where <TOKEN> is the Cloudflare Global API Key, that you can find instructions on how to find that key here: https://support.cloudflare.com/hc/en-us/articles/200167836-Where-do-I-find-my-Cloudflare-API-key-

That’s it. Now the flow is fully automated, and everything works as intended.

Experiment succeeded.

Links:

• https://wordpress.com/pricing/
• https://en.support.wordpress.com/custom-design/#frequently-asked-questions
• https://jekyllrb.com/
• https://jamstack.org/
• https://github.com/automata/awesome-jamstack