Imagine you're using Symfony and Doctrine to access a database. You are using prepared statements to prevent any SQL injection problems.

There are two main ways of doing this - and they disagree about how positional variables should be specified.

Data Retrieval And Manipulation

Here's a fairly trivial SQL statement with a couple of variables:

$sql = "SELECT `userID` FROM `users` WHERE `firstname` LIKE ? AND `surname` LIKE ?";
$stmt = $conn->prepare($sql);
$stmt->bindValue(1, $user_input_1st_name);
$stmt->bindValue(2, $user_input_2nd_name);
$results = $stmt->executeQuery();

Pretty easy, right? Write your SQL as normal, but place ? where you want user supplied variables to be. This uses bindValue() to set the variables in the query.

The approach using question marks is called positional, because the values are bound in order from left to right to any question mark found in the previously prepared SQL query. That is why you specify the position of the variable to bind into the bindValue() method Doctrine: Data Retrieval And Manipulation

Query Builder

Doctrine also offer an SQL Query Builder - it looks like this:

$queryBuilder = $conn->createQueryBuilder();
$queryBuilder
    ->select("userID")
    ->from("users")
    ->where("firstname LIKE ? AND surname LIKE ?")
    ->setParameter(0, $user_input_1st_name)
    ->setParameter(1, $user_input_2nd_name);
$results = $queryBuilder->executeQuery();

Notice the difference? Yes! setParameter() is zero based!

The numerical parameters in the QueryBuilder API start with the needle 0. SQL Query Builder

Why the difference?

Because the universe hates you, I guess?

Solving the issue

I've sent a pull request to make the documentation clearer. In the meantime, both methods accept named parameters.

$sql = "SELECT `userID` FROM `users` WHERE `firstname` LIKE :first";
$stmt->bindValue("first", $user_input_1st_name);

...

$queryBuilder
    ->select("userID")
    ->from("users")
    ->where("firstname LIKE :first")
    ->setParameter("first", $user_input_1st_name)

I hope that helps remove some confusion for future users. Even if it's only me!

It's a short and readable paper with some jaw-dropping anecdotes. Like the man who immediately got a pay rise after his transition, despite working in exactly the same job as before; women were on a lower pay scale...

At a basic level you can see why, when computer memory was measured in tens of kilobytes, it made sense to say male==0 and female==1. Why waste precious bits on something which could only ever be binary? Why create an option to change a data field which is immutable? Why design a schema which would allow a woman to be married to another woman?

And yet, even with those constraints, people were able to change their "official" gender within the database. Oh, sure, there were all sorts of cludges (both technical and political) - but it was possible.

The paper sparked four main thoughts for me.

There's No Such Thing As Immutable

For all the talk of Blockchain solving the world's issues (🤣) sometimes it is necessary to "rewrite history". People make mistakes. Assumptions change. Knowledge improves. Lots of facts, it turns out, are matters of perspective.

A really good example of this is time. I don't mean pesky things like timezones and leap seconds. I mean that, due to general relativity, one second on the moon is not equal to one second on Earth. How does your time-ordered database cope with that?

You might very well live in a culture where divorce is impossible, or where sexual consent cannot ever be revoked, or where a person can only be married to one other person at a time. But these are all societal conventions which are liable - and indeed likely - to change.

I'm almost tempted to say that the boolean type shouldn't exist in modern databases!

Diverse Teams Build Better Products

I don't know how many computer programmers in the 1960s were part of the LGBTQ+ community. And I don't know how accepting their colleagues would have been of them.

Perhaps you have read and memorised every single one of the Falsehoods Programmers Believe About... lessons. But surely it is more efficient to build a team who are empowered enough to confidently correct their colleagues' incorrect assumptions about how the world is arranged?

We bake rigid assumptions into our designs not out of malign intent (usually) but because we're ignorant. That's only shameful if we refuse to listen to other people's experiences.

Computers Serve Humans - not the other way around

Most of us have been forced to lie to a computer at one time or another. Perhaps it is a system which insists that you must have a US-style ZIP code. Or that your name must be longer that three characters. Or that you don't have an apostrophe in your email address. Or that your wife is Mrs, not Ms.

I know for sure that you've filled in a paper form where the boxes were too small and you've had to decide how to truncate your data.

Why? Because people have designed a schema which doesn't account for the variety in the world.

Today's constraints aren't tomorrow's

As I said at the start, it's understandable that designers designed around the constraints they faced. But these days, we have an awareness of the likely progress of technology.

It's said that the Apollo Moon landings were only possible because the designers skated to where the puck was going to be. They made reasonable assumptions about what technology was going to be developed in the future.

Yes, we should try and build things which perform well on existing and historic hardware. But we can't ignore the fact that tomorrow's computers will be smaller, faster, cheaper, and more efficient.

Does it make sense to store a human's name as:

CREATE TABLE people (
  name VARCHAR(32) CHARACTER SET latin1
);

Probably not. Disk space is cheap and getting cheaper. Perhaps people of the future will have names consisting of 500 emoji? Or perhaps people with "exotic" Unicode characters will want to use our services.

Oh, I'm sure there will be a performance hit if every column is essentially unlimited. But that's an argument to design better database engines - not to limit human expression.

Read More

You can read "Hacking the Cis-tem" in the IEEE or, if that's not available to you, read the pre-print.

Imagine that you work at a company which sells widgets. Each widget has a unique serial number. The number is a fixed length, and can contain leading zeros. That is, the following are all valid identifiers:

• 00001
• 01010
• 12345

What data type would you use to store these data in a database?

This is one of those strong opinions, weakly held. I'm not sure there's a right answer to it. A quick survey on Twitter⁰ was inconclusive:

Terence Eden is on Mastodon

@edent

You want to record a fixed-length number in your database (i.e. always 5 digits long).
The number can have leading zeros (e.g. 00123).
What data type do you use?

---

Numeric with ZEROFILL: (61)
61
Varchar or Text: (94)
94
Something else (what?): (12)
12

---

Here, I attempt to provide some of the opinions on both sides of the argument. Feel free to supply your own arguments.

It isn't a number

Can you do maths on it? If not, it isn't a number. It's a text string that happens to only contain numbers.

For example, asking "what is the average of these three serial numbers?" is a meaningless question. "What is this serial number divided by two?" again - that has an answer with no semantic content.

Counterpoint

What if the serial has some semantic content in it? For example even numbers were made in one factory, odds in another. Or prime numbers are demonstration units. It might be easier to grab information from a number type. But it's probably better to pull that data into separate fields.

It isn't guaranteed to always be a number

If this field was for the item's price, it's is always going to be a number. Sure, it might be a float or an int - but it'll be a number. Same if this was the item's height, width, or weight. Those values can only be numbers.

But there's nothing inherently "numbery" about a serial number. At any point, the boss could recommend adding Greek letters to it.

Given we can't do maths on it, and the "number" doesn't have any semantic content, there's no need to artificially restrict our database to a "number" type.

Counterpoint

The same is broadly true of any constraint. The maximum length could change. The validation rules might be updated. Modern databases need to be able to cope with changes in business requirements.

Technical Constraints

I don't know of any number type which stores leading zeros. Please enlighten me if I'm wrong. That means any data retrieved from the database has to be formatted before displaying it to the user.

That also means that searching the database requires data to be pre-formatted.

Counterpoint

All data should go through validation and sanitation before being displayed to the end user. Numbers aren't special in that regard. Similarly, data should be carefully checked before being searched for.

Numbers get misformatted

We all know how Microsoft Excel will look at any number and try to interpret it as a date. It also has a tendency to strip leading zeros. And some formatters will automatically add thousands separators to any number they see. Keeping the data as text reduces the risk of this happening.

Counterpoint

Excel will mangle anything that looks even vaguely like a number. Storing as text does not guarantee that it will be interpreted as text.

OK - I think that's the majority of the argument for not treating this as a number. What are the arguments on the other side?

Incorrect Data

If we allow text in this field, what happens if someone types a letter O rather than a number 0? Having this be an int prevents these errors creeping in. Yes, have checks at the front end, but this provides defence in depth.

Counterpoint

As above. Data should be checked by the application before submitting to the database. The database should be checking the data before storing it.

Searching and Sorting

Suppose we want to get all widgets with a serial number >= a specific ID. A number type is much better than string for those sorts of operations.

Counterpoint

This again assumes that the IDs have some semantic content. For example, 00012 was manufactured after 00008. This may or may not be the case. Such operations are best performed on a field like "Manufactured Date".

It's more efficient

It is quicker and computationally cheaper to store integers rather than text. Searching is faster, disk space requirements are lower.

Counterpoint

It isn't the 1970s. We're not paying per bit. Unless we're storing billions of rows, or working on constrained hardware, this isn't a practical concern for most users.

Human Usability

It looks like a number. A human, on encountering one of these IDs is going to assume it is a number. Anything which makes it harder for humans to understand is going to cause problems.

Counterpoint

Things like phone numbers look like integers, but they aren't. Momentary human confusion is preferable to mangled or imprecise data.

So now what?

I lean to the side that says that this is a string with specific constraints. Namely /^[0-9]{5}$/ (please supply your own regex meme).

When a user enters a new serial number, it should be checked that it meets the constraints. If it doesn't, refuse to submit it until the user has formatted it correctly. When submitted, the database should check against the constraints, and refuse to accept non-matching strings.

The speed of searching and sorting is not meaningfully degraded by storing as a string.

But, I'm very aware that I could be wrong. This is my strong opinion, but if you can supply a better argument, I will drop my weak grip on it.

Chaotic Good

Daniel Knell

@danielknell

Replying to @edent@edent 5x bigint fields, one for each digit.

---

You trust the Government, you voted for them, you and your friends have nothing to hide with regards to your sexuality.

But! Shock horror! After creating the database, the Government loses the election and the homophobes at UKIP get in to power!

Now they have a database of every gay in the village, and can harass then, try to "cure" them, or make their lives a living hell.

Far fetched? Not really. With Cameron's inane web filtering plan, the "black boxes" in ISPs which can record every click you make, and the selling of the your NHS details to private parties, we're in a situation where a malicious government could cause serious damage to us.

The security expert Bruce Schneier wrote a wonderful article for CNN on how the existing surveillance state is leading to disastrous breaches of our private information. He concludes by saying:

It's bad civic hygiene to build technologies that could someday be used to facilitate a police state.

-- Bruce Schneier on CNN

We have to be careful that the apparatus we build cannot easily be misused for evil purposes. Sure, even an innocuous toaster can be weaponised if someone is willing enough, but we should not fall into the trap of making systems which can easily be turned against the people.

It's probably sensible to build a database of which car belongs to which owner - it has an important civil use and would be hard to abuse (although not impossible).

Should we have a national database of, say, religious beliefs? Almost instinctively the answer is no. The memories of fascist dictators haunt our collective consciousness. We have seen countless times how race and religious identity become death penalties. We wouldn't countenance it.

Civic hygiene isn't about saying we distrust our current government - it's about not trusting the next government.

Ahem...

When I first started shkspr.mobi it was intended to be an easy way to get Shakespeare on your phone. At that time, there were no mobile formatted texts of his plays and sonnets, so I had to create them. Finding Shakespeare's works in a suitable format for conversion wasn't too hard - but it meant lots of crufty code to read text files line-by-line. Yuck.

A few years later, I stumbled across Open Source Shakespeare. The project grew out of Eric Johnson's MA thesis. It's a remarkably good idea with only one minor problem. The database it uses is Microsoft Access.

MS Access, as a database, could best be described as

deformed, crooked, old and sere, ill faced, worse bodied, shapeless everywhere, vicious, ungentle, foolish, blunt, unkind, stigmatical in making, worse in mind

(Comedy of Errors, Act IV, Scene II)

There are a few Open Source Shakespeare projects on GitHub, but they don't seem very practical.

So, naturally, I've decided to create my own version of Shakespeare's works - in MySQL :-)

This is what it looks like: You can download it from GitHub.

I've stripped out a lot of the extraneous stuff from the original version - word counts, etc. So it should be a fairly lean database which is easy to use. I'm not a database professional, so I would be grateful if you could suggest any improvements. Either using this blog's comment form or on GitHub.

There are four tables

Paragraphs

This is where the main body of text is. A typical row will look like this

• WorkID: hamlet
• ParagraphID: 639015
• ParagraphNum: 3427
• CharID: hamlet
• PlainText: Has this fellow no feeling of his business, that he sings atngrave-making?
• Act: 5
• Scene: 1

Works

This is what translates the "WorkID" into something human readable - plus some extra metadata

• WorkID: hamlet
• Title: Hamlet
• LongTitle: Tragedy of Hamlet, Prince of Denmark, The
• Date: 1600
• GenreType: Tragedy

Character

This is what translates the CharID into a human readable name and description

• charID: hamlet
• CharName: Hamlet
• Abbrev: Ham
• Works: Tragedy of Hamlet, Prince of Denmark, The
• Description: son of the former king and nephew to the present king

Chapters

This gives the setting for each Act and Scene.

• WorkID: hamlet
• ChapterID: 18893
• Act: 5
• Scene: 1
• Description: Elsinore. A churchyard.

What's Next?

The next steps for the project are fairly obvious:

1. Write some high level example code to show people how to use the database.
2. Make shkspr.mobi a showcase site which runs off the database.
3. Fix any bugs and inconsistencies that people find.

You can download the Shakespeare MySQL Database from GitHub.