Imagine you're using Symfony and Doctrine to access a database. You are using prepared statements to prevent any SQL injection problems.

There are two main ways of doing this - and they disagree about how positional variables should be specified.

Data Retrieval And Manipulation

Here's a fairly trivial SQL statement with a couple of variables:

$sql = "SELECT `userID` FROM `users` WHERE `firstname` LIKE ? AND `surname` LIKE ?";
$stmt = $conn->prepare($sql);
$stmt->bindValue(1, $user_input_1st_name);
$stmt->bindValue(2, $user_input_2nd_name);
$results = $stmt->executeQuery();

Pretty easy, right? Write your SQL as normal, but place ? where you want user supplied variables to be. This uses bindValue() to set the variables in the query.

The approach using question marks is called positional, because the values are bound in order from left to right to any question mark found in the previously prepared SQL query. That is why you specify the position of the variable to bind into the bindValue() method Doctrine: Data Retrieval And Manipulation

Query Builder

Doctrine also offer an SQL Query Builder - it looks like this:

$queryBuilder = $conn->createQueryBuilder();
$queryBuilder
    ->select("userID")
    ->from("users")
    ->where("firstname LIKE ? AND surname LIKE ?")
    ->setParameter(0, $user_input_1st_name)
    ->setParameter(1, $user_input_2nd_name);
$results = $queryBuilder->executeQuery();

Notice the difference? Yes! setParameter() is zero based!

The numerical parameters in the QueryBuilder API start with the needle 0. SQL Query Builder

Why the difference?

Because the universe hates you, I guess?

Solving the issue

I've sent a pull request to make the documentation clearer. In the meantime, both methods accept named parameters.

$sql = "SELECT `userID` FROM `users` WHERE `firstname` LIKE :first";
$stmt->bindValue("first", $user_input_1st_name);

...

$queryBuilder
    ->select("userID")
    ->from("users")
    ->where("firstname LIKE :first")
    ->setParameter("first", $user_input_1st_name)

I hope that helps remove some confusion for future users. Even if it's only me!

Soundex is a clever algorithm for reducing a string of characters into a string which roughly represents its pronunciation. The "roughly" is key here. We could just search the database for identical entries, but users often mistype entries. So comparing sounds is a good way to gloss over those mistakes.

This gives us the Soundex of every inscription in the database:

SELECT soundex(`inscription`) FROM `benches` WHERE 1 

Which results in:

|soundex(`inscription`)                               | 
|=====================================================|
|D531323161545613216323                               |
|J51616565324126143216316531414                       |
|D232624132635253263532154                            |
|H6432534215361312345252342525321625342323153423163...| 
|A2316215635                                          |

Effectively, it's like a fuzzy hash for text.

So, here's how to get a list of all the Soundexs which have duplicates:

SELECT `inscription`, SOUNDEX(`inscription`), COUNT(SOUNDEX(`inscription`))
    FROM    `benches`
    WHERE `published` = 1
    GROUP BY SOUNDEX(`inscription`)
    HAVING COUNT(SOUNDEX(`inscription`)) > 1
    LIMIT 0 , 1024"

Which I can then use to produce a list of duplicates:

And I can then get a list of all benches with a specific Soundex:

SELECT `benchID`, `inscription`, `address`
    FROM    `benches`
    WHERE  SOUNDEX(`inscription`) = "A123"
    AND  `published` = 1

In this case, I can see that they're similar but not identical - so I don't need to merge them.

I started off with a thousand possible duplicates and, after going through each of them, merged a couple of hundred dupes. That was a fun weekend!

Other Strategies

There were a couple of other things I thought of, and then discarded, to deal with dupes.

Ask the user on upload

This is probably the simplest. If a new upload has a similar Soundex to a nearby bench, ask the user if it a duplicate. But, actually, that's fraught with complexity. It puts a lot of pressure on a new user to get it right. And, frankly, we're a tiny community which needs all the users it can get. So I decided to put the pressure back on me, the admin.

There are quite a few benches near each other which have identical inscriptions. So asking a user to check which bench they meant could also be complicated.

Which leads on to...

Check proximity

There are lots of benches near each other. That's understandable. But calculating how near any two benches are is a bit more complex. They're stored in the database with separate latitude and longitude values. So, we can use the Haversine formula to find all benches within, say, 250 metres of position 1.23,4.56:

SELECT
(
    6371 * ACOS(COS(RADIANS(1.23)) *
    COS(RADIANS(`latitude`)) *
    COS(RADIANS(`longitude`) -
    RADIANS(4.56)) +
    SIN(RADIANS(1.23)) *
    SIN(RADIANS(`latitude`)))
)
AS distance, benchID, latitude, longitude, inscription, published
FROM benches
WHERE published = true AND present = true
HAVING distance < .25
ORDER BY distance

That's fine - but when combined with Soundex, it becomes much more complex. Effectively you then have to group similar sounds by geographic closeness. And, frankly, I really can't be bothered!

If you'd like to help out, the code to OpenBenches is open source.

Gravatars are based on the MD5 hash of a user's email. For some reason, the plugin had overwritten the avatar field with the text http://identicon

This MySQL query finds all the comment IDs which have that dodgy text:

SELECT 
   comment_id FROM `wp_commentmeta` 
   WHERE `meta_key` LIKE 'avatar' 
   AND `meta_value` LIKE 'http://identicon'

Using a SubQuery we can find all the email addresses for those comments - and generate an MD5 for them:

SELECT 
   comment_author_email, 
   MD5(comment_author_email) AS md5 
   FROM `wp_comments` 
   WHERE comment_id IN
      (SELECT 
         comment_id FROM `wp_commentmeta` 
         WHERE `meta_key` LIKE 'avatar' 
         AND `meta_value` LIKE 'http://identicon')

A good first step, but how to get all those MD5s into URls and then back into the database?

First, they need to be concatanated.

SELECT 
   comment_id, 
   CONCAT("https://www.gravatar.com/avatar/", MD5(comment_author_email)) AS gravatar
   FROM `wp_comments` 
   WHERE comment_id IN
      (SELECT 
         comment_id FROM `wp_commentmeta` 
         WHERE `meta_key` LIKE 'avatar' 
         AND `meta_value` LIKE 'http://identicon')

Let's put that all together in one query:

UPDATE wp_commentmeta
   JOIN wp_comments
   ON wp_commentmeta.comment_id = wp_comments.comment_ID
   SET meta_value = CONCAT("https://www.gravatar.com/avatar/", MD5(comment_author_email))
   WHERE meta_key LIKE 'avatar'
   AND meta_value LIKE "http://identicon"

Run that, and all your broken Gravatars will be regenerated.

Thanks to Ed Jefferson for his help with this!

Ahem...

When I first started shkspr.mobi it was intended to be an easy way to get Shakespeare on your phone. At that time, there were no mobile formatted texts of his plays and sonnets, so I had to create them. Finding Shakespeare's works in a suitable format for conversion wasn't too hard - but it meant lots of crufty code to read text files line-by-line. Yuck.

A few years later, I stumbled across Open Source Shakespeare. The project grew out of Eric Johnson's MA thesis. It's a remarkably good idea with only one minor problem. The database it uses is Microsoft Access.

MS Access, as a database, could best be described as

deformed, crooked, old and sere, ill faced, worse bodied, shapeless everywhere, vicious, ungentle, foolish, blunt, unkind, stigmatical in making, worse in mind

(Comedy of Errors, Act IV, Scene II)

There are a few Open Source Shakespeare projects on GitHub, but they don't seem very practical.

So, naturally, I've decided to create my own version of Shakespeare's works - in MySQL :-)

This is what it looks like: You can download it from GitHub.

I've stripped out a lot of the extraneous stuff from the original version - word counts, etc. So it should be a fairly lean database which is easy to use. I'm not a database professional, so I would be grateful if you could suggest any improvements. Either using this blog's comment form or on GitHub.

There are four tables

Paragraphs

This is where the main body of text is. A typical row will look like this

• WorkID: hamlet
• ParagraphID: 639015
• ParagraphNum: 3427
• CharID: hamlet
• PlainText: Has this fellow no feeling of his business, that he sings atngrave-making?
• Act: 5
• Scene: 1

Works

This is what translates the "WorkID" into something human readable - plus some extra metadata

• WorkID: hamlet
• Title: Hamlet
• LongTitle: Tragedy of Hamlet, Prince of Denmark, The
• Date: 1600
• GenreType: Tragedy

Character

This is what translates the CharID into a human readable name and description

• charID: hamlet
• CharName: Hamlet
• Abbrev: Ham
• Works: Tragedy of Hamlet, Prince of Denmark, The
• Description: son of the former king and nephew to the present king

Chapters

This gives the setting for each Act and Scene.

• WorkID: hamlet
• ChapterID: 18893
• Act: 5
• Scene: 1
• Description: Elsinore. A churchyard.

What's Next?

The next steps for the project are fairly obvious:

1. Write some high level example code to show people how to use the database.
2. Make shkspr.mobi a showcase site which runs off the database.
3. Fix any bugs and inconsistencies that people find.

You can download the Shakespeare MySQL Database from GitHub.