Mechanics of software

Classification of logical data models implemented by contemporary DBMS

2026-02-12T11:23:00+01:00

Apart from relational model, many contemporary DBMS implement different data models, including extensions of relational ones. Most of DBMS implement also multiple data models, and are so called “multi-model”. However, there is no commonly used classification of actual data models that is a disorientating factor for database users: engineers, students, teachers, analysts, etc. This article contains common terms and definitions as well as the history of earlier model classification proposed by other experts in the database domain since the 1970s. Developing the graph-based and set-based classification approach, and classical hierarchy-network-relational triad, the article proposes an improved two-axis data model classification including new semi-structured ones as well as examples of DBMS implementing these models.

Full article text on ResearchGate

Data migration checklist

2025-05-27T12:01:00+02:00

Points to check before any data migration.

Process constraints:
- optimal and maximum allowed time to perform all operations (e.g. weekend, 48 hours)
- rollback case: partial, full?
- resume migration from the last failed operation or restart all?
- monitoring: logs, UI, console output, technical data...
Source data:
- formats and descriptions
- data volume estimation (bytes, rows, tables, tuples...)
- who is responsible to explain data logic in terms of business logic?
- data access:
  - directly to DBMS - how to connect?
  - exported files - consider the time to export fresh data to file, file access permissions
  - is data cleanup required?
Destination data:
- database access (how to, and what is minimal permissions level required e.g. DB owner)
- database model (schema) with description
- who is responsible to explain data logic in terms of business logic?
- any constraints that may affect data import: storage limits, triggers, replication...
ETL/ELT:
- allowed/recommended software (e.g. SQL Server Integration Services, custom .NET or Python application...)
- running environment, required access and authorizations, data connections (CLI components)
- transformations: rules to validate expected data integrity and consistency
Dev environment:
- almost same software configurations as for production environment: software versions, OS, CPU architecture
- infrastructure may be different
Test environments:
- same configuration/infrastructure as for production environment
- performance testing is "the must"

Short history of information technologies

2025-04-30T11:51:00+02:00

Ancient world

1950s – Compilers will make programmers obsolete
1960-70s - 3rd generation programming languages will reduce programmer shortages

...

Middle Ages

1980s - 4rd generation programming languages and CASE tools will definitely reduce programmer shortages
1980s – client server SQL DBMS will solve mainframe “vendor lock”
1990s – 3-tiers architecture will solve DBMS “vendor lock”
1996 – XML and SOAP will fix interoperability between tiers and app server “vendor lock”
1996 – Java will fix code portability (write once – run everywhere)
1997 – CORBA will solve all integration issues
1998 – MS Java extension will fix Sun Java poor performance
2000 – .NET will fix Java “vendor lock”

...

Modern History

2001 – Enterprise Service Buses (ESB) will solve CORBA complexity and "vendor lock"
2005 – SOA will fix the challenges ESB introduced
2006 – NoSQL will overcome relational database limits
2008 – REST APIs will simplify SOA and fix SOAP performance
2009 – "SQL" over NoSQL databases will fix NoSQL poor querying
2015 – Microservices will save us from SOA complexity (and monoliths)
2016 – Docker will fix microservice headaches
2018 – Kubernetes will fix Docker headaches
2020 – Service Mesh will make Kubernetes manageable
2022 – Serverless programming will abstract away Service Mesh
2024 – AI Observability will bring clarity to Serverless chaos
2025 - AI will replace programmers
^^ YOU ARE HERE ^^

Source: my LinkedIn

SQL Server as key-value store versus Redis

2024-12-08T18:38:00+01:00

Any considerations and improvements of this test are welcome.

Motivation

Redis is a performant solution for application level cache. However, when a system have already running instances of SQL Server, do you really need Redis in addition?

The main argument I expected to hear is the Redis performance. Let's check if this statement has some proofs.

Environment

As an SQL Server certified expert I could up its performance results for specific cases. However, it is not the goal. Let's take both products "out of the box".

For the fair results I just take:

SQL Server container: version 2022 (16.x) under Linux Ubuntu 22.04 LTS
Redis container: version 7.1.4 "latest" under Debian 12 (Bookworm slim)
run these two containers on a laptop PC

Host PC:

CPU i7-1370P: 7 cores, hyper-threading is off
RAM 32 GB
OS Ubuntu 22.04 LTS
Docker version 20.10.21, build 20.10.21-0ubuntu1~22.04.3

Testing scenario

Both databases contain one million "key-value" pair. The key is of UUID (GUID) type, and the values are randomly generated strings of 100 characters of length.

The test performs random access concurrent readings from N threads in cycles of M requests.

Example of SQL Server query (pseudo code):

v = Exec("SELECT v FROM kv WHERE k = <UUID>").Scalar()

Example of Redis query:

v = db.StringGet(uuid)

Testing program

Full sources are available on my GitHub repository.

Initializing of testing data is written as SQL Server script 00_prepare_data.sql that:

creates a test SQL Server database
creates a key-value table
fill table with test data (data is persistent)

Testing program is written on C#, framework version is 8.0.404.

Build:

$ dotnet build -c Release

Running parameters

$ ./MSSQLvsRedis [Number of threads] [requests per thread]

Number of threads is 10 by default
Requests per thread is 1000 by default

Pay attention that the very first run will takes a time: the program is loading data to Redis store. BTW, SQL Server loading of these data is much faster (bulk load), but this is not the test subject.

Results

For the random access concurrent reading test SQL Server is generally faster than Redis.

It does not mean that "SQL Server is better", but apparently you need other technical arguments to approve Redis usage in addition to existing DBMS. Remember also that SQL Server performance is not gratis, and other "free" (free of license charges) DBMS would be much less performant.

You may take a look on the full results in the results.xlsx file.

Notes

There are several ways to improve SQL Server performance:

Redis is expected to be running inside a container, whereas SQL Server is running usually on the dedicated server (even hardware server).
GUID type for the keys is not the best choice, and its performance is about 30% lower than for integer value keys. Testing GUIDs are also fragmented, there is an option to use newsequentialid() function to avoid it.
SQL server performance under the Linux OS may be lower than under "native" Windows regardless the SQL OS implemented inside the DBMS.

DBMS market share 2011-2021

2023-05-03T12:48:00+02:00

The "Top 3 of vendors" is still here but new cloud-based DBMS are rising. Open source solutions like PostgreSQL (Enterprise DB) or MySQL (MariaDB) are in stagnation.

List of user sessions in SQL

2023-04-01T16:51:00+02:00

The frequently asked problem on interviews.

You have a log of some user activities represented as a table. Every activity record has at least users ID and activity date/time values.

The session is a sequence of activities having less than N minutes between two log records. When the elapsed time is greater than N minutes, the session is terminated; the new one is started.

The question is how to calculate the number of session per user.

Solution

The commonly proposed solution is based on using the LAG() window function which may not be supported by some DBMS. Therefore, the "old school pure SQL" solution still seems to be simple and is running everywhere.

Suppose the simplified log table user_activities which has only two required columns: user_id and activity_dt.

CREATE TABLE user_activities (
    user_id integer NOT NULL,
    activity_dt datetime NOT NULL
);

Insert the test data; I've also marked session breaks in the comments.

INSERT INTO user_activities (user_id, activity_dt)
VALUES
    -- user 1
    (1, '2021-03-05 10:05:00'),
    (1, '2021-03-05 10:10:00'), -- break 1
    (1, '2021-03-05 10:20:00'), -- 2
    (1, '2021-03-05 20:05:00'),
    (1, '2021-03-05 20:05:00'),
    (1, '2021-03-05 20:10:00'),
    (1, '2021-03-05 20:15:00'),
    (1, '2021-03-05 20:20:00'),
    (1, '2021-03-05 20:25:00'),
    (1, '2021-03-05 20:30:00'), -- 3
    (1, '2021-03-05 20:40:00'),
    -- user 2
    (2, '2021-03-05 10:05:00'),
    (2, '2021-03-05 10:08:00'),
    (2, '2021-03-05 10:10:00'), -- 1
    (2, '2021-03-05 10:12:00'),
    (2, '2021-03-05 10:20:00'), -- 2
    (2, '2021-03-05 20:10:00'),
    (2, '2021-03-05 20:15:00'),
    (2, '2021-03-05 20:20:00'),
    (2, '2021-03-05 20:25:00'),
    (2, '2021-03-05 20:30:00'), -- 3
    (2, '2021-03-05 20:40:00'), -- 4
    (2, '2021-03-05 20:46:00')
;

Now we can get the session count using CTE to simplify the query only. Note that I use the SQL Server specific function DATEDIFF which should be replaced by counterparts for other DBMS. E.g. TIMESTAMPDIFF() for MySQL, EXTRACT(EPOCH FROM... for PostgreSQL and so on.

Suppose also that the maximum delay breaking a session is 5 minutes.

WITH periods AS (
    SELECT
        ua1.user_id,
        ua1.activity_dt AS dt1,
        (
            SELECT MIN(ua2.activity_dt)
            FROM user_activities ua2
            WHERE ua2.user_id = ua1.user_id AND ua2.activity_dt > ua1.activity_dt
        ) AS dt2
    FROM user_activities ua1
)
SELECT user_id, COUNT(1) AS session_count
FROM periods
WHERE DATEDIFF(minute, dt1, dt2) > 5 -- max delay
GROUP BY user_id
;

Result

user_id     session_count
----------- -------------
1           3
2           4

(2 rows affected)

Should database only store data?

2022-11-13T22:51:00+01:00

Very old subject I remember to discuss in the middle of 1990s... But some people still says that a clean architecture should remove all business logic from the database.

Let's start from referential and domain integrity rules (constraints). Are they business ones? Yes, of course. The e-mail column should be unique in the table because of application logic. Any adepts of "clean approach" prefer to not mention this.

Step two. Were DBMS created in 1960s to only store data? No they weren't of corse. You can use any simple tools to store data like "flat file" but the final goal is to be able to extract from these data for user needs.

So you need a query language; otherwise welcome to the pre-SQL world where any data extraction should be implemented as a small program like Python over NoSQL does today.

A usual database query is a declarative language program; its Python counterpart will count 10 times more lines of code. However, the complex queries may have many dozens or even small hundreds of lines. Is it business logic code? Yes, of course.

Finally, the real world typical case.

Let's take Mongo - it's fashionable, stylish and youthful
Hurrah! We have loaded all data easily. But, damn it, there's something wrong with the queries...
Switching to Postgres... Although it is not youthful, but still fashionable, and claimed as free
Hooray! Queries became easier to write in this powerful human-friendly language now. But damn it, why is it so slow? Do Oracle and Microsoft want our money for their query optimizers and storages developed over decades?
No, we don't have a money to buy a DBMS from Big-3, so we will have to move many parts of the calculations from queries and stored procedures to the code of services written on Node.js...

Why use interface instead of pimpl

2022-11-13T22:51:00+01:00

The opaque pointer (pimpl) idiom has been inherited from C language where it is used to encapsulate implementation details. However, both old-school and "modern" C++ dispense you from writing some ugly code, and allow to use interfaces with object factories.

The example of implementation with both pimpl and interface approach is as follows.

pimpl.hpp: pimpl implementation header

#include <string>
#include <memory>

using namespace std;

class book
{
public:
    book(const string& title);
    book(const string& title, const string& author_name, const int page_count);
    // If you use pimpl with unique_ptr, you need to declare a destructor
    // which should be implemented in '*.cpp' file
    ~book();
public:
    // Main interface
    string title() const;
    string author_name() const;
    int page_count() const;
private:
    class book_impl; // Defined and implemented in '*.cpp' file
    unique_ptr<book::book_impl> m_impl; // Opaque pointer
};

using book_ptr = unique_ptr<book>;

pimpl.cpp: implementation

#include "pimpl.hpp"
// Here we include some headers i.e. of third-party libraries
// which are required for implementation only
// Once being published, the 'pimpl.hpp' does not depend on them

class book::book_impl
{
public:
    book_impl(const string& name, const string& author_name, const int page_count)
        : m_title(name), m_author_name(author_name), m_page_count(page_count)
    {}
    // Main interface implementation
    string title() const {
        return m_title;
    }

    string author_name() const {
        return m_author_name;
    }

    int page_count() const {
        return m_page_count;
    }
private:
    string m_title;
    string m_author_name;
    int m_page_count;
};

// Now we need to code the redirection of ALL calls
book::book(const string& title)
    : book(title, "Unknown", 1)
{}

book::book(const string& title, const string& author_name, const int page_count)
    : m_impl(make_unique<book_impl>(title, author_name, page_count))
{}

book::~book()
{}

string book::title() const {
    return m_impl->title();
}

string book::author_name() const {
    return m_impl->author_name();
}

int book::page_count() const {
    return m_impl->page_count();
}

no_pimpl.hpp: interface and factory header

#include <string>
#include <memory>

using namespace std;

class book_intf
{
public: // Main interface
    virtual string title() const = 0;
    virtual string author_name() const = 0;
    virtual int page_count() const = 0;
public:
    virtual ~book_intf() {}
};

using book_intf_ptr = unique_ptr<book_intf>;

class book_factory
{
public:
    static book_intf_ptr create(const string& title);
    static book_intf_ptr create(const string& title, const string& author_name, const int page_count);
};

no_pimpl.cpp: implementation

#include "no_pimpl.hpp"

class book_impl : public book_intf
{
public:
    book_impl(const string& title)
        : book_impl(title, "Unknown", 1)
    {}
    book_impl(const string& title, const string& author_name, const int page_count)
        : m_title(title), m_author_name(author_name), m_page_count(page_count)
    {}
public: // Main interface implementation
    string title() const override {
        return m_title;
    }

    string author_name() const override {
        return m_author_name;
    }

    int page_count() const override {
        return m_page_count;
    }
private:
    string m_title;
    string m_author_name;
    int m_page_count;
};

// Book factories
book_intf_ptr book_factory::create(const string& title) {
    return move(make_unique<book_impl>(title));
}

book_intf_ptr book_factory::create(const string& title, const string& author_name, const int page_count) {
    return move(make_unique<book_impl>(title, author_name, page_count));
}

As you can see, redirection of calls is not required when using interfaces, so you do not need to define functions signatures twice with copy-past coding.

Running both:

#include "pimpl.hpp"
#include "no_pimpl.hpp"
#include <memory>
#include <iostream>

int main(int, char**) {
     auto book1 = std::make_unique<book>("Modern old-school C++", "Gang of thousands", 255);
     cout << "Title: " << book1->title()
         << "\nAuthor: " << book1->author_name()
         << "\nPages: " << book1->page_count()
         << endl;

    book_intf_ptr book2 = book_factory::create("Modern old-school C++", "Gang of thousands", 255);
    cout << "Title: " << book2->title()
        << "\nAuthor: " << book2->author_name()
        << "\nPages: " << book2->page_count()
        << endl;
    return 0;
}

About factories

We can define factories at least several different ways:

separate factory class with static methods
separate factory class with non-static methods (need when additional initializations/settings are required before create an instance)
static method within interface class (may be a small harm when derive a new interface from this one)
namespace functions (simplest but minimally abstract)

Get all subsets from a set

2022-02-01T21:33:00+01:00

How to extract all combinations (unordered subsets) from a given set in C++?

Let us estimate the count before. Suppose N is the size of a given set, and K is the number of elements in the subset. The count of all combinations (unordered subsets) of size K is

C(N, K) = N! / (K! * (N - K)!)

Then the count of all subsets is

C(N, 1) + C(N, 2) + ... + C(N, N) = 2^N - 1

So we expect to produce 7 subsets from a set of 3 elements, 15 subsets from 4-elements set, and so on.

#include <iostream>
#include <vector>

using set_t = std::vector<int>;
using subsets_t = std::vector<set_t>;

void make_combinations(const set_t& data, const size_t index, const set_t& current_subset, subsets_t& subsets)
{
    for (size_t i = index; i < data.size(); i++)
    {
        set_t subset = current_subset;
        subset.push_back(data[i]);
        subsets.push_back(subset);
        make_combinations(data, i + 1, subset, subsets);
    }
}

subsets_t get_subsets(const set_t& data)
{
    subsets_t subsets;
    make_combinations(data, 0, set_t(), subsets);
    return subsets;
}

void print_subsets(const subsets_t& subsets)
{
    std::cout << "Subset count: " << subsets.size() << std::endl;
    for (const auto& subset : subsets)
    {
        for (const auto value : subset)
        {
            std::cout << value << " ";
        }
        std::cout << std::endl;
    }
    std::cout << std::endl;
}

int main()
{
    subsets_t subsets = get_subsets({ 1, 2, 3, 4 });
    print_subsets(subsets);
    return 0;
}

Result

Subset count: 15
1
1 2
1 2 3
1 2 3 4
1 2 4
1 3
1 3 4
1 4
2
2 3
2 3 4
2 4
3
3 4
4

All expressions of 123456789 and signs +/- which value is 100

2021-09-30T21:09:00+02:00

Suppose a string of ordered cyphers 123456789. You can insert signs "+" and "-" between any cyphers to make a correct arithmetical expression. The problem is to find all expressions which sum is 100.

This problem may be interesting for dynamic script languages having the function of a string expression evaluation.

E.g. solving in bash

for i in {,-}1{,+,-}2{,+,-}3{,+,-}4{,+,-}5{,+,-}6{,+,-}7{,+,-}8{,+,-}9; do (($i == 100)) && echo $i; done

Result

123+45-67+8-9
123+4-5+67-89
123-45-67+89
123-4-5-6-7+8-9
12+3+4+5-6-7+89
12+3-4+5+67+8+9
12-3-4+5-6+7+89
1+23-4+56+7+8+9
1+23-4+5+6+78-9
1+2+34-5+67-8+9
1+2+3-4+5+6+78+9
-1+2-3+4+5+6+78+9

The standard SQL has no evaluation functions. However, the recursive query approach can help.

Solution in ANSI SQL

WITH signs AS (
    SELECT '' AS s, 0 AS sv
    UNION
    SELECT '+' AS s, 1 AS sv
    UNION
    SELECT '-' AS s, -1 AS sv
),
curr (sv, c, expr, num, val) AS (
    SELECT 0 AS sv, 0 AS c, CAST('' AS varchar(20)) AS expr, 0 AS num, 0 AS val
    UNION ALL
    SELECT
        signs.sv AS sv, c + 1 AS c,
        CAST(expr + signs.s + CAST((c + 1) AS varchar(1)) AS varchar(20)) AS expr,
        CASE
            WHEN signs.sv = 0
                THEN num * 10 + (CASE WHEN num < 0 THEN - (c + 1) ELSE c + 1 END)
            ELSE signs.sv * (c + 1)
        END AS num,
        CASE WHEN signs.sv = 0 THEN val ELSE val + num END AS val
    FROM curr CROSS JOIN signs
    WHERE curr.c < 9 AND NOT (c = 0 AND signs.sv = 1)
)
SELECT expr, val + num AS total
FROM curr
WHERE c = 9 AND val + num = 100

Result

expr                 total
-------------------- -----------
-1+2-3+4+5+6+78+9    100
1+2+3-4+5+6+78+9     100
1+2+34-5+67-8+9      100
1+23-4+5+6+78-9      100
1+23-4+56+7+8+9      100
12-3-4+5-6+7+89      100
12+3-4+5+67+8+9      100
12+3+4+5-6-7+89      100
123-4-5-6-7+8-9      100
123-45-67+89         100
123+4-5+67-89        100
123+45-67+8-9        100

(12 rows affected)