In their post "How I Reversed Amazon's Kindle Web Obfuscation Because Their App Sucked" they describe the process of spoofing a web browser, downloading a bunch of JSON files, reconstructing the obfuscated SVGs used to draw individual letters, and running OCR on them to extract text.

There were a few problems with this approach.

Firstly, the downloader was hard-coded to only work with the .com site. That fix was simple - do a search and replace on amazon.com with amazon.co.uk. Easy!

But the harder problem was with the OCR. The code was designed to visually centre each extracted glyph. That gives a nice amount of whitespace around the character which makes it easier for OCR to run. The only problem is that some characters are ambiguous when centred:

When I ran the code, lots of full-stops became midpoints, commas became apostrophes, and various other characters went a bit wonky.

That made the output rather hard to read. This was compounded by the way line-breaks were treated. Modern eBooks are designed to be reflowable - no matter the size of your screen, lines should only break on a new paragraph. This had forced linebreaks at the end of every displayed line - rather than at the end of a paragraph.

So I decided to fix it.

A New Approach

I decided that OCRing an entire page would yield better results than single characters. I was (mostly) right. Here's what a typical page looks like after de-obfuscation and reconstruction:

As you can see - the typesetting is good for the body text, but skew-whiff for the title. Bold and italics are preserved. There are no links or images.

Here's how I did it.

Extract the characters

As in the original code, I took the SVG path of the character and rendered it as a monochrome PNG. Rather than centring the glyph, I used the height and width provided in the glyphs.json file. That gave me a directory full of individual letters, numbers, punctuation marks, and ligatures. These were named by fontKey (bold, italic, normal, etc).

Create a blank page

The page_data_0_4.json has a width and height of the page. I created a white PNG with the same dimensions. The individual characters could then be placed on that.

Resize the characters

In the page_data_0_4.json each run of text has a fontKey - which allows the correct glyph to be selected. There's also a fontSize parameter. Most text seems to be (the ludicrously precise) 19.800001. If a font had a different size, I temporarily scaled the glyph in proportion to 19.8.

Each glyph has an associated xPosition, along with a transform which gives X and Y offsets. That allows for indenting and other text layouts.

The characters were then pasted on to the blank page.

Once every character from that page had been extracted, resized, and placed - the page was saved as a monochrome PNG.

OCR the page

Tesseract 5 is a fast, modern, and reasonably accurate OCR engine for Linux.

Running tesseract page_0022.png output -l eng produced a .txt file with all the text extracted.

For a more useful HTML style layout, the hOCR output can be used: tesseract page_0022.png output -l eng hocr

Or, a PDF with embedded text: tesseract page_0022.png output -l eng pdf

Mistakes

OCR isn't infallible. Even with a high resolution image and a clear font, there were some errors.

• Superscript numerals for footnotes were often missing from the OCR.
• Words can run together even if they are well spaced.
• Tesseract can recognise bold and italic characters - but it outputs everything as plain text.

What's missing?

Images aren't downloaded. I took a brief look and, while there are links to them in the metadata, they're downloaded as encrypted blobs. I'm not clever enough to do anything with them.

The OCR can't pick out semantic meaning. Chapter headings and footnotes are rendered the same way as text.

Layout is flat. The image of the page might have an indent, but the outputted text won't.

What's next?

This is very far from perfect. It can give you a visually similar layout to a book you have purchased from Amazon. But it won't be reflowable.

The text will be reasonably accurate. But there will be plenty of mistakes.

You can get an HTML layout with hOCR. But it will be missing formatting and links.

Processing all the JSON files and OCRing all the images is relatively quick. But tweaking and assembling is still fairly manual.

There's nothing particularly clever about what I've done. The original code didn't come with an open source software licence, so I am unable to share my changes - but any moderately competent programmer could recreate this.

Personally, I've just stopped buying books from Amazon. I find that Kobo is often cheaper and their DRM is easy to bypass. But if you have many books trapped in Amazon - or a book is only published there - this is a barely adequate way to liberate it for your personal use.

But here's the thing with the LCP DRM scheme; they do give you the key! As I've written about previously, LCP mostly relies on the user entering their password (the key) when they want to read the book. Oh, there's some deep cryptographic magic in the background but, ultimately, the key is sat on your computer waiting to be found. Of course, cryptography is Very Hard™ which make retrieving the key almost impossible - so perhaps we can use a different technique to extract the unencrypted content?

One popular LCP app is Thorium. It is an Electron Web App. That means it is a bundled browser running JavaScript. That also means it can trivially be debugged. The code is running on your own computer, it doesn't touch anyone else's machine. There's no reverse engineering. No cracking of cryptographic secrets. No circumvention of any technical control. It doesn't reveal any illegal numbers. It doesn't jailbreak anything. We simply ask the reader to give us the content we've paid for - and it agrees.

Here Be Dragons

This is a manual, error-prone, and tiresome process. This cannot be used to automatically remove DRM. I've only tested this on Linux. It must only be used on books that you have legally acquired. I am using it for research and private study.

This uses Thorium 3.1.0 AppImage.

First, extract the application:

./Thorium-3.1.0.AppImage --appimage-extract

That creates a directory called squashfs-root which contains all the app's code.

The Thorium app can be run with remote debugging enabled by using:

./squashfs-root/thorium --remote-debugging-port=9223 --remote-allow-origins=*

Within the Thorium app, open up the book you want to read.

Open up Chrome and go to http://localhost:9223/ - you will see a list of Thorium windows. Click on the link which relates to your book.

In the Thorium book window, navigate through your book. In the debug window, you should see the text and images pop up.

In the debug window's "Content" tab, you'll be able to see the images and HTML that the eBook contains.

Images

The images are the full resolution files decrypted from your ePub. They can be right-clicked and saved from the developer tools.

Files

An ePub file is just a zipped collection of files. Get a copy of your ePub and rename it to whatever.zip then extract it. You will now be able to see the names of all the files - images, css, fonts, text, etc - but their contents will be encrypted, so you can't open them.

You can, however, give their filenames to the Electron app and it will read them for you.

Images

To get a Base64 encoded version of an image, run this command in the debug console:

fetch("httpsr2://...--/xthoriumhttps/ip0.0.0.0/p/OEBPS/image/whatever.jpg") .then(response => response.arrayBuffer())
  .then(buffer => {
    let base64 = btoa(
      new Uint8Array(buffer).reduce((data, byte) => data + String.fromCharCode(byte), '')
    );
    console.log(`data:image/jpeg;base64,${base64}`);
  });

Thorium uses the httpsr2 URl scheme - you can find the exact URl by looking at the content tab.

CSS

The CSS can be read directly and printed to the console:

fetch("httpsr2://....--/xthoriumhttps/ip0.0.0.0/p/OEBPS/css/styles.css").then(response => response.text())
  .then(cssText => console.log(cssText));

However, it is much larger than the original CSS - presumably because Thorium has injected its own directives in there.

Metadata

Metadata like the NCX and the OPF can also be decrypted without problem:

fetch("httpsr2://....--/xthoriumhttps/ip0.0.0.0/p/OEBPS/content.opf").then(response => response.text())
  .then(metadata => console.log(metadata));

They have roughly the same filesize as their encrypted counterparts - so I don't think anything is missing from them.

Fonts

If a font has been used in the document, it should be available. It can be grabbed as Base64 encoded text to the console using:

fetch("httpsr2://....--/xthoriumhttps/ip0.0.0.0/p/OEBPS/font/Whatever.ttf") .then(response => response.arrayBuffer())
  .then(buffer => {
    let base64 = btoa(
      new Uint8Array(buffer).reduce((data, byte) => data + String.fromCharCode(byte), '')
    );
    console.log(`${base64}`);
  });

From there it can be copied into a new file and then decoded.

Text

The HTML of the book is also visible on the Content tab. It is not the original content from the ePub. It has a bunch of CSS and JS added to it. But, once you get to the body, you'll see something like:

<body>
    <section epub:type="chapter" role="doc-chapter">
        <h2 id="_idParaDest-7" class="ct"><a id="_idTextAnchor007"></a><span id="page75" role="doc-pagebreak" aria-label="75" epub:type="pagebreak"></span>Book Title</h2>
        <div class="_idGenObjectLayout-1">
            <figure class="Full-Cover-White">
                <img class="_idGenObjectAttribute-1" src="image/cover.jpg" alt="" />
            </figure>
        </div>
        <div id="page76" role="doc-pagebreak" aria-label="76" epub:type="pagebreak" />
        <section class="summary"><h3 class="summary"><span class="border">SUMMARY</span></h3> 
        <p class="BT-Sans-left-align---p1">Lorem ipsum etc.</p>
    </section>

Which looks like plain old ePub to me. You can use the fetch command as above, but you'll still get the verbose version of the xHTML.

Putting it all together

If you've unzipped the original ePub, you'll see the internal directory structure. It should look something like this:

├── META-INF
│   └── container.xml
├── mimetype
└── OEBPS
    ├── content.opf
    ├── images
    │   ├── cover.jpg
    │   ├── image1.jpg
    │   └── image2.png
    ├── styles
    │   └── styles.css
    ├── content
    │   ├── 001-cover.xhtml
    │   ├── 002-about.xhtml
    │   ├── 003-title.xhtml
    │   ├── 004-chapter_01.xhtml
    │   ├── 005-chapter_02.xhtml
    │   └── 006-chapter_03.xhtml
    └── toc.ncx

Add the extracted files into that exact structure. Then zip them. Rename the .zip to .epub. That's it. You now have a DRM-free copy of the book that you purchased.

BONUS! PDF Extraction

LCP 2.0 PDFs are also extractable. Again, you'll need to open your purchased PDF in Thorium with debug mode active. In the debugger, you should be able to find the URl for the decrypted PDF.

It can be fetched with:

fetch("thoriumhttps://0.0.0.0/pub/..../publication.pdf") .then(response => response.arrayBuffer())
  .then(buffer => {
    let base64 = btoa(
      new Uint8Array(buffer).reduce((data, byte) => data + String.fromCharCode(byte), '')
    );
    const blob = new Blob([buffer], { type: "application/pdf" });
    const link = document.createElement("a");
    link.href = URL.createObjectURL(blob);
    link.download = "publication.pdf"; // filename for saving
    link.click();
    URL.revokeObjectURL(link.href);
  });

Copy the output and Base64 decode it. You'll have an unencumbered PDF.

Next Steps

That's probably about as far as I am competent to take this.

But, for now, a solution exists. If I ever buy an ePub with LCP Profile 2.0 encryption, I'll be able to manually extract what I need from it - without reverse engineering the encryption scheme.

Ethics

Before I published this blog post, I publicised my findings on Mastodon. Shortly afterwards, I received a LinkedIn message from someone senior in the Readium consortium - the body which has created the LCP DRM.

They said:

Hi Terence, You've found a way to hack LCP using Thorium. Bravo!

We certainly didn't sufficiently protect the system, we are already working on that.

From your Mastodon messages, you want to post your solution on your blog. This is what triggers my message.

From a manual solution, others will create a one-click solution. As you say, LCP is a "reasonably inoffensive" protection. We managed to convince publishers (even big US publishers) to adopt a solution that is flexible for readers and appreciated by public libraries and booksellers.

Our gains are re-injected in open-source software and open standards (work on EPUB and Web Publications).

If the DRM does not succeed, harder DRMs (for users) will be tested.

I let you think about that aspect

I did indeed think about that aspect. A day later I replied, saying:

Thank you for your message.

Because Readium doesn't freely licence its DRM, it has an adverse effect on me and other readers like me.

• My eReader hardware is out of support from the manufacturer - it will never receive an update for LCP support.
• My reading software (KOReader) have publicly stated that they cannot afford the fees you charge and will not be certified by you.
• Kobo hardware cannot read LCP protected books.
• There is no guarantee that LCP compatible software will be released for future platforms.

In short, I want to read my books on my choice of hardware and software; not yours.

I believe that everyone deserves the right to read on their platform of choice without having to seek permission from a 3rd party.

The technique I have discovered is basic. It is an unsophisticated use of your app's built-in debugging functionality. I have not reverse engineered your code, nor have I decrypted your secret keys. I will not be publishing any of your intellectual property.

In the spirit of openness, I intend to publish my research this week, alongside our correspondence.

Their reply, shortly before publication, contained what I consider to be a crude attempt at emotional manipulation.

Obviously, we are on different sides of the channel on the subject of DRMs.

I agree there should be many more LCP-compliant apps and devices; one hundred is insufficient. KOReader never contacted us: I don't think they know how low the certification fee would be (pricing is visible on the EDRLab website). FBReader, another open-source reading app, supports LCP on its downloadable version. Kobo support is coming. Also, too few people know that certification is free for specialised devices (e.g. braille and audio devices from Hims or Humanware).

We were planning to now focus on new accessibility features on our open-source Thorium Reader, better access to annotations for blind users and an advanced reading mode for dyslexic people. Too bad; disturbances around LCP will force us to focus on a new round of security measures, ensuring the technology stays useful for ebook lending (stop reading after some time) and as a protection against oversharing.

You can, for sure, publish information relative to your discoveries to the extent UK laws allow. After study, we'll do our best to make the technology more robust. If your discourse represents a circumvention of this technical protection measure, we'll command a take-down as a standard procedure.

A bit of a self-own to admit that they failed to properly prioritise accessibility!

Rather than rebut all their points, I decided to keep my reply succinct.

As you have raised the possibility of legal action, I think it is best that we terminate this conversation.

I sincerely believe that this post is a legitimate attempt to educate people about the deficiencies in Readium's DRM scheme. Both readers and publishers need to be aware that their Thorium app easily allows access to unprotected content.

I will, of course, publish any further correspondence related to this issue.

At the risk of sounding like an utter corporate stooge, I think it is a relatively inoffensive and technically interesting DRM scheme. Primarily because, once you've downloaded your DRM-infected book, you don't need to rely on an online server to unlock it.

How does it work?

When you buy⁰ a book, your vendor sends you a .lcpl file. This is a plain JSON file which contains some licencing information and a link to download the ePub.

Here's a recent one of mine (truncated for legibility):

{
    "issued": "2025-03-04T12:34:56Z",
    "encryption": {
        "profile": "http://readium.org/lcp/profile-2.0",
        "content_key": {
            "algorithm": "http://www.w3.org/2001/04/xmlenc#aes256-cbc",
            "encrypted_value": "+v0+dDvngHcD...qTZgmdCHmgg=="
        },
        "user_key": {
            "algorithm": "http://www.w3.org/2001/04/xmlenc#sha256",
            "text_hint": "What is your username?",
            "key_check": "mAGgB...buDPQ=="
        }b
    },
    "links": [
        {
            "rel": "publication",
            "href": "https://example.com/96514dea-...-b26601238752",
            "type": "application/epub+zip",
            "title": "96514dea-...-b26601238752.epub",
            "length": 14364567,
            "hash": "be103c0e4d4de...fb3664ecb31be8"
        },
        {
            "rel": "status",
            "href": "https://example.com/api/v1/lcp/license/fdcddcc9-...-f73c9ddd9a9a/status",
            "type": "application/vnd.readium.license.status.v1.0+json"
        }
    ],
    "signature": {
        "certificate": "MIIDLTCC...0faaoCA==",
        "value": "ANQuF1FL.../KD3cMA5LE",
        "algorithm": "http://www.w3.org/2001/04/xmldsig-more#ecdsa-sha256"
    }
}

Here's how the DRM works.

1. Your client downloads the ePub from the links section.
2. An ePub is just a zip file full of HTML files, the client unzips it.
   • The metadata and cover image are not encrypted - so you can always see the title and cover. All the rest - HMTL, images, fonts, etc - are encrypted with AES 256 CBC.
3. The .lcpl file is placed in the META-INF directory and renamed license.lcpl.
4. A new ePub is created by re-zipping the files together.

When your client opens the encrypted ePub, it asks you for a password. If you don't know it, you get the hint given in the LCPL file. In this case, it is my username for the service where I bought the book.

The password is used by Readium's super-secret BLOB to decrypt the file. You can then read the book.

But here's the nifty thing, the encrypted file is readable by any certified app. I used the LCPL to download the book in two different readers. I unzipped both of them and they were bit-for-bit identical. I copied the book from one reader to another, and it was read fine. I built my own by downloading the ePub and manually inserting the licence file - and it was able to be opened by both readers.

Apps and Certification

In order for this to work, the app needs to be certified and to include a binary BLOB which does all the decryption. Readium have a list of readers which are available, and there are plenty for all platforms.

On Linux, I tried Thorium and FBReader. Both were absolutely fine. For my eInk Android, I used FBReader Premium (available for free if you don't have Google Play installed). Again, it was a decent reading experience.

I took the file created by Thorium on Linux, copied it to Android, set the Android offline, typed in my password, and the book opened.

Open Source and DRM

To be fair to Readium, they publish a lot of Open Source code and the specification seems well documented.

But the proprietary BLOB used for the decryption is neither libre -

Thorium Reader supports LCP-protected publications via an additional software component which is not available in this open-source codebase

Nor gratis -

Our pricing is structured into tiers based on a company’s revenue

What's the worst that could happen with this DRM?

Ultimately, our fear of DRM comes down to someone else being able to control how, when, and even if we can read our purchased books. Could that happen here?

I'm going to go with a cautious maybe.

Positives

Once downloaded, the ePub is under your control. Back it up on a disk, store it in the cloud, memorise the bytes. It is yours and can't be forcibly deleted. You can even share it with a friend! But you'd have to tell them the book's password which would make it trivially linkable to you if it ever got shared widely.

At the moment, any LCP book reading app will open it. Even if your licence is somehow revoked, apps don't need to go online. So there is no checking for revocation.

ePub is an open standard made up of zipped HTML, CSS, images, and fonts. An unencrypted ePub should be readable far into the future. LCP is a (paid for) ISO Standard which is maintained by a foundation which is primarily run by an EU non-profit. So, hopefully, the DRM scheme will also be similarly long-lived.

Because the underlying book is an ePub, it should have the same accessibility features as a normal ePub. No restrictions on font-sizes, text-to-speech, or anything similar.

Privacy. The BLOB only checks with the issuer of the book whether the licence is valid. That's useful for library books where you are allowed to borrow the text for a specific time. If you bought books from a dozen sources, there's no central server which tracks what you're reading across all services.

Downsides

Will the proprietary BLOB work in the future? If it never gets ported to Android 2027 or TempleOS, will your books be rendered unreadable on your chosen platform?

The LCPL file contains dates and signatures related to the licence. Perhaps the BLOB is instructed to check the licence after a certain period of time. Will your books refuse to open if the BLOB hasn't gone online for a few years?

If you forget your password, you can't open the book. Thankfully, the LCPL does contain a "hint" section and a link back to the retailer. However, it's up to you to securely store your books' passwords.

The book seller knows what device you're reading on. When you load the LCPL file into a reader, the app downloads the ePub and sends some data back to the server. The URl is in the status section of the LCPL file. After opening the file on a few apps, mine looked like:

{
    "id": "fdcddcc9-...-f73c9ddd9a9a",
    "status": "active",
    "updated": {
        "license": "2025-03-04T12:34:56Z",
        "status": "2025-03-09T20:20:20Z"
    },
    "message": "The license is in active state",
    "links": [
        {
            "rel": "license",
            "href": "https://example.com/lcp/license/fdcddcc9-...-f73c9ddd9a9a",
            "type": "application/vnd.readium.lcp.license.v1.0+json"
        }
    ],
    "events": [
        {
            "name": "Thorium",
            "timestamp": "2025-03-04T15:49:37Z",
            "type": "register",
            "id": "7d248cae-...-c109b887b7dd"
        },
        {
            "name": "FBReader@framework",
            "timestamp": "2025-03-08T22:36:26Z",
            "type": "register",
            "id": "46838356-...-73132673"
        },
        {
            "name": "FBReader Premium@Boyue Likebook-K78W",
            "timestamp": "2025-03-09T14:54:26Z",
            "type": "register",
            "id": "e351...3b0a"
        }
    ]
}

So the book seller knows the apps I use and, potentially, some information about the platform they're running on. They also know when I downloaded the book. They may also know if I've lent a book to a friend.

It is trivial to bypass this just by downloading the ePub manually and inserting the LCPL file as above.

DRM Removal

As I've shown before, you can use OCR to rip an eBook. Take a bunch of screenshots, extract the text, done. OK, you might lose some of the semantics and footnotes, but I'm sure a bit of AI can solve that. The names of embedded fonts can easily be read from the ePub. But that's not quite the same as removing the DRM and getting the original ePub.

When the DeDRM project published a way to remove LCP 1.0, they were quickly hit with legal attacks. The project removed the code - although it is trivial to find on 3rd party sites. Any LCP DRM removal tool you can find at the moment is only likely to work on Basic Encryption Profile 1.0.

There are now multiple different encryption profiles:

In 2024, the EDRLab Encryption Profile 1.0 was superseded by 10 new profiles, numbered “2.0” to “2.9”. Every LCP license provider chooses one randomly and can easily change the profile.

If I'm reading the source code correctly¹, the user's password is SHA-256 hashed and then prefixed with a secret from the LCP code. That is used as the decryption key for AES-256-CBC.

I'm sure there's some digital trickery and obfuscation in there but, at some point, the encrypted ePub is decrypted on the user's machine. Maybe it is as simple as grabbing the binary and forcing it to spit out keys. Maybe it takes some dedicated poking about in memory to grab the decrypted HTML. Given that the key is based on a known password, perhaps it can be brute-forced?

I'll bet someone out there has a clever idea. After all, as was written by the prophets:

trying to make digital files uncopyable is like trying to make water not wet²

What we do have is a Blockbuster card and, between us, just enough cash to rent a newly released movie. Eight of us pile into the local Blockbuster and begin to scavenge the shelves. DVDs have yet to appear in our sleepy town, so we hold up chunky VHS boxes for appraisal.

"The English Patient?" "Rubbish."

"Trainspotting?" "Seen it."

"The Full Monty?" "My mum won't let me watch that."

"Titanic?" "Oooooh! Go on then!"

I took the empty carcass up to the desk. The incredibly worldly-wise 19 year old looked at us disdainfully. "Sorry mate, out of stock."

You see, hard as it may be to believe, the local Blockbuster franchise had underestimated the popularity of Titanic. They'd purchased a dozen copies to rent out. Once those walked out the door, that was it. We could ask them to ring us when a tape was returned - but we'd be at the bottom of a long waiting list.

So we leafed through a few display racks and went home with the kid's film about a genie, "Shazaam". It wasn't very good, as I recall.

If only Netflix were like that! Imagine if you tried to watch Stranger Things and were told that too many people were streaming the first episode. Netflix will notify you when there's a slot available. In the meantime, why not watch "Clarissa Explains It All"?

Wouldn't that be brilliant! Netflix could restrict demand, make things artificially scarce, and just generally make a worse user experience for everyone.

Wait... no... that'd be shit!

And yet, that's exactly how digital library books work in the UK!

Look, perhaps a Netflix for books wouldn't work; it's harder to read than it is to stick on TV in the background. But there's a healthy library scene in the UK. Authors get paid when people borrow their books. So why are libraries restricted to only loaning out a limited number of copies at once?

There's a part of me - a very small part - which can see the utility in this. If everyone could borrow the latest Dan Brown smash-hit all at once, would they ever explore any further? When you're told "Sorry, we don't have that - but you might like..." it could expand your reading horizons.

But that's a weak argument. With an endless supply of books, people can choose what they want to read, when they want to read it.

There is no digital scarcity. We need to rid the world of this analogue thinking.

I tried to explain to them that DRM always fails; you can't make data which can't be copied. I explained that artificial scarcity was harmful. They didn't care.

But, their proposed solution was intriguing. And, with their kind permission, I'm posting it here. To be clear, I don't think this is good but I think it is vaguely interesting.

Interlacing

Back in the olden days, video was often interlaced. That is, the TV transmitter would broadcast the odd lines first, then the even lines. Your TV would draw the odd line then, in the millisecond before they faded away from the cathode-ray tube, draw the even lines.

Could we do the same with images?

Let's take this image of a Total Legitimate Bored Ape™

With a scrap of Python, we can knock this into two interlaced images:

from PIL import Image
image = Image.open("ape.png")
pixels = image.load()
image.mode = "RGBA"

for y in range(image.height) :
   if (y%2 == 1) :
      for x in range(image.width) :
         pixels[x,y] = (0,0,0,0)

image.save("ape1.png")

for y in range(image.height) :
   if (y%2 == 0) :
      for x in range(image.width) :
         pixels[x,y] = (0,0,0,0)

image.save("ape2.png")

Which gives us:

Using JavaScript the first frame can be rendered onto a <canvas> element for a millisecond, wiped, and then the second displayed. Repeat.

Here's a demo of it in action - I kinda like the retro effect!

⚠️ Strobe Effect Warning

If you right click on it, you'll only get half the image. If you take a screenshot, you'll only get half the image. FOOLPROOF DRM!!!!!!

Expanding the complexity

Of course, you can scramble an image before unscrambling it to the canvas and all sort of other tricks. You could have dozens of images rendering to the canvas in sequence. You could draw individual pixels programmatically. You could...

None of it matters of course. If you send pixels to a computer screen, they can be copied. It is possible to slow people down - but all it takes is for one person to work out how your scheme works and then it crumbles to dust.

As I said, I have a soft-spot for the shimmery effect. And I think it is kinda fun - in a code-golf way - to design obfuscation schemes.

But using anything like this for a form of digital restrictions management is daft.

Those photographs are "Crown Copyright". For photographs created before 1st June 1957, the copyright expires after 50 years.

Recently, the organisation "Historic England" started sharing high-resolution copies of these photos on a nifty interactive map.

But there were two problems.

Firstly, they claimed that the photographs were still under copyright. This (no doubt inadvertent) mistake was pointed out to them and was eventually corrected.

Andy Mabbett

@pigsonthewing

It seems that, without announcement, @HistoricEngland have dropped the claim that copyright-expired pre-June 1957 RAF images are still in copyright, from their Aerial Photo Explorer pic.x.com/uceK7qzQxd

---

The second, and to my mind more troubling, problem is that the photos were "protected" using SmartFrame's Digital Restrictions Management.

SmartFrame has some useful features - it allows for high-resolution photos to be loaded "zoomed out" in lower resolution. The user can then zoom in on a portion which then gets loaded as higher resolution. That's it really. SmartFrame's main selling point is that it "brings robust image control". AKA, it uses DRM to prevent users from downloading images.

It also promises "Complete image protection". This is nonsense. If you transmit an image to a user, the user can copy that image.

Here's how easy it is to download the images which SmartFrame claims to protect.

Screenshots

The obvious flaw in SmartFrame is that users can take screenshots of the high-resolution image. Zoom in, screenshot, pan left, another screenshot, repeat.

Of course, stitching together all those images is a bit of a pain. But perfectly possible to automate if you wanted to.

Canvas Chunks

The way SmartFrame works is by loading small "chunks" of the image and then drawing then on a <canvas> element.

In your browser's network inspector, you'll see each 256x256 sub-image loading. The images are not encrypted, so they can be saved directly. Again, it is a manual and tedious process to scrape them all and then stitch them together.

Inspecting the network requests shows that they all use the same Accept Header of authorization/wndeym9ajvin,*/* - that appears to be common across multiple SmartFrame instances. Bad form of them to reuse that key!

Canvas Access

It's fairly easy to download anything drawn onto a <canvas> element by running:

var c = document.getElementsByClassName("stage") 
c[0].toDataURL() 

However, SmartFrame have overloaded the .toDataURL() function - so it produces a warning when you try that. It's simple enough to disable their JS once the image has loaded.

Of course, the <canvas> is smaller than the full resolution image - so you may need to manually increase its size first.

It's also possible to simply right-click on the <canvas> in the inspector and copy the Base64 representation of the image:

Putting it all together

I am indebted to Stuart Langridge for connecting all the dots. He has written and fully documented some code which is, essentially:

1. Grab the canvas
2. Resize it
3. Wait several seconds for the image chunks to fully load onto the canvas
4. Turn the canvas into a Data URL
5. Download the data

It looks something like this:

var container = document.querySelector("div.articlePage.container");
container.style.width="6000px";
container.style.maxWidth="6000px";

setTimeout(()=>{
  var stage = document.querySelector("canvas.stage");
  var url = document.createElement("canvas").toDataURL.call(stage);
  var a = document.createElement("a");
  a.href = url;
  a.download = "liberated.png";
  a.click();
}, 3000);

And that's it. A user can paste a dozen lines of Javascript into their browser's console and get a full-resolution PNG.

Warnings

This technique should only be used to download images which are free of copyright restrictions.

Companies should be careful before buying a DRM solution and ensure that it is fit for purpose. SmartFrame really isn't suitable as sold. Despite its grandiose claims of "Super-strong encryption" and "Multi-layered theft-prevention" - it took less than weekend to bypass.

It is possible that SmartFrame will update their systems to defeat this particular flaw. But, thankfully, DRM can never work effectively. You can't give users a locked box and a key - then expect them to only unlock the box under the "right" circumstances. As Bruce Schneier once said:

trying to make digital files uncopyable is like trying to make water not wet.

This is a quick-and-dirty way to grab images of the pages and convert them to a standard PDF using Linux. There's a lot more you can do to make the end book more useful, but this'll get you started

Lots of Screen Shots

With a USB cable plugged into my phone and laptop, I wrote a horrible little bash script:

#!/bin/bash
for i in {00001..00555}; do
   adb exec-out screencap -p > $i.png
   adb shell input tap 1000 2000
   sleep 1s
done
echo All done

This runs a loop 555 times. Takes a screenshot, names it for the loop number with padded zeros, taps the bottom right of the screen, then waits for a second to ensure the page has refreshed. Slow and dull, but works reliably.

Images range from 200KB to 2MB depending on complexity. Back them up before doing the next bit.

Cropping

The screenshots are all 1080x2160. But the page only takes up part of that. The top left corner is at 50x432 and the bottom right is at 1028x1726.

This command crops all the images. It is destructive, so make sure you have a backup.

mogrify -crop 978x1294+50+432 +repage *.png

It's also useful to trim the images to remove any whitespace from the borders. That makes a smaller file size.

mogrify -trim *.png

Images can be shrunk with:

pngquant *.png

PDF and OCR

Sticking all the images together into a single PDF is pretty easy:

convert *.png +repage output.pdf

The +repage option keeps the aspect ratio of the trimmed image.

But there's no text to search. There are a bunch of OCR programs on Linux, I like PDF Sandwich:

pdfsandwich -rgb -nopreproc output.pdf

That'll get you a colour PDF with OCR'd text embedded in it. The text is "sandwiched" behind the image of the page, so you can't see it but can search for it.

You can also use OCRmyPDF which may result in a smaller file:

ocrmypdf -l eng output.pdf output_ocr.pdf

And that's it. I now have a searchable PDF which I can read on any device.

What have we learned?

DRM on textbooks is an annoyance. For computer science books, it's little more than a fig-leaf.

That's a situation which continues over a decade later. If you watch iPlayer on your laptop, you're stuck with 720p quality. If you want 1080p and above, you need a specially certified device.

Well, that's what everyone thought!

A few weeks ago, I found this curious forum post which said

Yes, iPlayer 1080p is possible, not as simple as just using one of these webUI tools but its certainly possible with youtube-dl and a good look around.. audio_eng_1=128000-video=12000000.m3u8

There wasn't much information about it that I could find. And then I stumbled onto this GitHub issue for youtube-dl.

With a bit of faffing around, I was able to patch my YouTube downloader (/usr/local/lib/python3.9/dist-packages/youtube_dl/extractor/bbc.py) with this code:

for fmt in formats:
   if '-video=5070000.m3u8' in fmt.get('url', ''):
      formats = [
         {'url': fmt['url'].replace('-video=5070000.m3u8', '-video=12000000.m3u8'),
          'height': 1080, 'vbr': 12000,
          'abr': fmt.get('abr'),
          'protocol': fmt.get('protocol'),
          'preference': fmt.get('preference'),
          'format_id': fmt.get('format_id', 'hack') + '-really12M',
          }, fmt]

(To be clear, this is a horrible hack. It changes the iPlayer download function so it only downloads 1080p. It also doesn't rename the file properly once reconstructed, so you'll have to do that yourself. This is a quick hack - not for production!)

And, with that, I was able to download iPlayer in 1080p.

Sadly there's no surround sound - AAC stereo only.

Learning from history

Back in 2008 (!) I was blogging about the BBC's ridiculous decision to restrict iPlayer streaming to iPhones. And in 2012 I blogged about how the BBC's insistence on using Flash had misfired. The BBC (and, to be fair, the video rights-holders) are desperately trying to avoid their content being shown at high quality on "unsanctioned" devices.

I get that they don't want to lose revenue. But anyone with a USB stick can record HD off air. 4K rips of every movie appear on the Internet 5 minutes after they're released. Trying to restrict who can view your content is a losing battle. It only takes one person to figure it out, and then the genie is out of the bottle.

What's the technical overhead of trying to manage this complicated infrastructure they've created? What is the tangible loss if someone watches a 4K stream from their browser rather than the app?

At some point, surely, media companies will have to realise that making easy access to their content is more powerful than blocking off every creative use. Right?

Bah. I'm off to count the pixels on the latest episode of Doctor Who.

libgourou is a free implementation of Adobe's ADEPT protocol used to add DRM on ePub files. It overcome the lacks of Adobe support for Linux platforms.

There are a few limitations, but nothing too serious:

• Only ePub is supported. No PDF
• Command line only
• Alpha quality software. It works - but is a bit finickity

You can compile the code yourself, but I just downloaded the pre-built binaries.

On Ubuntu / Pop I needed to sudo apt install libzip-dev.

Once you've unzipped the download, cd into it and run:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PWD

Then

./adept_activate -u youremail@example.com -p SuperStrongPassword

Use the same credentials as you do in Adobe Digital Editions. You can just use the -u flag and type your password in at the prompt - but I couldn't get that to work. The -p flag worked fine.

After a few seconds, you'll see: youremail@example.com fully signed and device activated in /home/you/libgourou_utils_0.3.1/./.adept

Then run: ./acsmdownloader -f whatever.acsm

After a few moments, you'll see: Created whatever.epub

That's it! Your file is downloaded and you can read the ePub.

Note: If you try to download a PDF, you'll get an error saying: Exception code : 0x5007 Message : whatever.epub

You can rename the file back to .pdf, but it will still be encrypted.

Sure, you can futz around with Docker, WINE, and old versions of Python - and if you're lucky, you might get a book out of it.

I wasn't quite so lucky. I wanted to see if I could download an ePub without using Adobe Digital Editions. Spoilers! I couldn't. But this is what I found.

The First ACSM file

When you buy an eBook, the vendor sends you a .acsm file. Adobe Content Server Message is a simple XML file. This is what it looks like and what each line means:

<fulfillmentToken fulfillmentType="buy" auth="user" xmlns="http://ns.adobe.com/adept">
  <distributor>urn:uuid:b479af7f-09bf-48f5-a6c6-...</distributor>
  <operatorURL>http://drm.firebrandtech.com/fulfillment</operatorURL>
  <transaction>67809895895d46aab...</transaction>
  <expiration>2021-07-19T18:55:59-04:00</expiration>

This states I've purchased the book, who distributes it, where the download server is, and some metadata

Next, metadata about the book:

  <resourceItemInfo>
    <resource>urn:uuid:77d79621-0e96-4f4f-8fee-...</resource>
    <resourceItem>0</resourceItem>
    <metadata>
      <dc:title xmlns:dc="http://purl.org/dc/elements/1.1/">The End of Bias</dc:title>
      <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Jessica Nordell</dc:creator>
      <dc:publisher xmlns:dc="http://purl.org/dc/elements/1.1/">Granta Publications</dc:publisher>
      <dc:identifier xmlns:dc="http://purl.org/dc/elements/1.1/">urn:ean:9781846276774</dc:identifier>
      <dc:format xmlns:dc="http://purl.org/dc/elements/1.1/">application/pdf</dc:format>
    </metadata>
    <licenseToken>
      <resource>urn:uuid:77d79621-0e96-4f4f-8fee-...</resource>
      <permissions>
        <display/>
        <excerpt/>
        <print/>
        <play/>
      </permissions>
    </licenseToken>
  </resourceItemInfo>

Finally, a digital signature:

  <hmac>rL4Ttn7ASp16Y4...=</hmac>
</fulfillmentToken>

The Second ACSM file

When you open the .acsm file in Adobe Digital Editions (ADE), the software creates a new .acsm file. This one is signed with the key that Adobe generates when you register with it. This is what the file looks like and what each line means:

First, information about me as a user, and my device:

<fulfill xmlns="http://ns.adobe.com/adept">
  <user>urn:uuid:99b1e32d-5b39-4968-8ef6-c5cb6a739ec8</user>
  <device>urn:uuid:ac790662-5da0-4075-aee2-99b3c4592619</device>
  <deviceType>standalone</deviceType>

Then a repeat of the original .acsm:

  <fulfillmentToken fulfillmentType="buy" auth="user">
    <distributor>urn:uuid:b479af7f-09bf-48f5-a6c6-...</distributor>
    <operatorURL>http://drm.firebrandtech.com/fulfillment</operatorURL>
    <transaction>67809895895d46...</transaction>
    <expiration>2021-07-19T18:55:59-04:00</expiration>
    <resourceItemInfo>
      <resource>urn:uuid:77d79621-0e96-4f4f-8fee-...</resource>
      <resourceItem>0</resourceItem>
      <metadata>
        <dc:title xmlns:dc="http://purl.org/dc/elements/1.1/">The End of Bias</dc:title>
        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Jessica Nordell</dc:creator>
        <dc:publisher xmlns:dc="http://purl.org/dc/elements/1.1/">Granta Publications</dc:publisher>
        <dc:identifier xmlns:dc="http://purl.org/dc/elements/1.1/">urn:ean:9781846276774</dc:identifier>
        <dc:format xmlns:dc="http://purl.org/dc/elements/1.1/">application/pdf</dc:format>
      </metadata>
      <licenseToken>
        <resource>urn:uuid:77d79621-0e96-4f4f-8fee-...</resource>
        <permissions>
          <display/>
          <excerpt/>
          <print/>
          <play/>
        </permissions>
      </licenseToken>
    </resourceItemInfo>

Then, a different HMAC:

    <hmac>rL4Ttn7ASp16...=</hmac>
  </fulfillmentToken>

And, finally, another digital signature

  <signature>znoLnR2A4wG1uE85oXPYzSZjTRNo76ctgfR8COY8nlNTCcPge9O6VeB5UY...=</signature>
</fulfill>

POSTING the file

Digital Editions then posts the new .acsm file to the DRM server mentioned. It replies with a location where the file can be downloaded from. In this case:

http://drm.firebrandtech.com/media/77d79621-0e96-4f4f-8fee-....pdf

That's literally just the resource's UUID as the filename. It can be downloaded and opened in ADE.

Signing the file yourself

So, if you've downloaded the first .acsm file and have your encryption keys extracted from ADE you can sign it yourself, right?

I dunno. I'm no good at reverse engineering binaries. I suspect so. ADE is a Shockwave / Flash app - so I doubt it has anything too taxing in there - but it is beyond my skillset.

Further Reading

• gedonder's notes on ACSM
• Karl Voit's rant about ADE

Do you ever see those daft email footers which say "Please consider the environment before printing this email." Like, who the fuck is still printing out their emails?

Anyway, a few years ago I went along to a blogging event where someone had printed out one of my blog posts. I was stunned. They'd stuck of loads of my posts (and other people's posts) on a mood board. Because I'm a digital fundamentalist, this pissed me off - what a waste of paper. Printing off a blog post is like making a radio broadcast of a sunset. Just pointless.

Luckily, there's a (very weak) form of DRM you can use to dissuade people from printing off your web pages:

@media print {
  html { display: none; }
}

CSS has Media Rules for printing. You're supposed to use it to remove background images, menu bars, and other things which aren't useful. In this case, I've just told the browser not to display any content when it tries to print.

You can go a little further. This stops the page from rendering when printed, and adds a handy error message:

@media print{
    body { display: none; }
    html::after {
        content: "Please consider the environment before printing this blog.";
    }
}

I've tested this, and it works in Firefox, Safari, and Chrome. It's trivial to defeat this technical protection mechanism if you're handy with developer tools - but I guess those aren't the people who are printing off the entire Internet.

Here's the problem:

Why am I getting the error "This content can not be displayed because your TV does not support HDCP 2.2."?

I have four pieces of kit in the mix, all of which claim to support HDCP's Digital Restrictions Management:

• Amazon Fire TV Stick - the new 4K version
• Toshiba 49U6863 - a 4K TV
• Denon AVR-X2500H - a 4K surround sound amp
• HDMI 2.0 cable - an 18Gbps rated cable

Easy, right? No!

Here's what does work: Fire -> TV If I plug the Fire Stick directly into the TV, it works! I get 4K HDR. So I'm confident the TV does support HDCP.

Here's what does not work:

Fire -> Denon -> HDMI cable -> TV Causes the Denon to display the "your TV does not support HDCP 2.2." error message when 4K content is played.

Is it the cable?

The HDMI cable claims to support 4K video. The Denon has a built in cable tester which confirms that the cable meets the "4K (6G)" specification.

The cable is short - under 2 meters - so shouldn't run into any latency issues.

Is it the TV's config?

I don't think so. There is an HDMI Full Range settings, which I have enabled.

I've set the HDMI input to "Advanced".

The Fire seems to think the TV is compatible, and lets me switch to 4K, even when it is going through the Denon.

To be clear, Fire -> Denon -> HDMI cable -> TV (Enhanced) does not show an error on the Fire but shows an error on the Denon about HDCP..

Is it the signal pathway?

Despite Denon's marketing material claiming that every port can do 4K HDR, their support pages tell a different story!

I've tried plugging the Fire stick into the closest HDMI port - but it made no difference.

Is it the amp's settings?

There are dozens of permutations of settings, but I think I've narrowed it down to the essentials.

I've turned off all video processing and set the amp to "bypass" all video processing from the HDMI video path.

I think this is where the problem lies - the 4K Signal Format setting.

If I set it to "Standard", I can watch 4K video without HDR.

As soon as I set it to "Enhanced", the Denon shows the HDCP error.

The Denon shows the TV as accepting 4K HDR.

Can you strip the DRM?

Possibly. Devices like Dr HDMI's HD Fury claim they can fix these problems. But I'm loathe to spend another €90 to fix a problem not of my own making.

Conclusion?

I have no clue! Either the Fire or the Denon are doing the HDCP handshake incorrectly. Or, perhaps, the TV doesn't handshake when connected to an amp.

You'd think it would be simple to pass a digital signal from one device to another, via a 3rd - but the paranoia of DRM means paying customers have to put up with all manner of user-hostile limitations.

Next steps?

• Both the TV and Amp were purchased from RicherSounds - so I might give them a call to see if they've any ideas.
• Amazon support don't think it is a problem with the Fire but will pass my comments onto their engineering team.
• I've contacted Toshiba - but they outsource their manufacturing to Vestel, so who knows if I'll ever get an answer.
• Denon have promised to contact me in 48 hours.

Is it my fault for being a media snob?

Well, probably. I should spend less time watching the picture quality, and more time watching the show.

But I'm an innocent customer. I've spent good money to watch high-def media and I'm being stymied because of DRM.

I'd love there to be an open source AV amp so I could actually be in control of what I watch.

If you've got any smart ideas for how to fix this - please drop me a comment!

Update

I returned the Denon thinking it was faulty and got the Pioneer VSX-933. Sadly, it also has the same error! So, next steps are to find another 4K HDR source to see if that works, or... I dunno!

I was taking a look at Google's "Family" proposition - which allows users to share their purchases with other family members. What I found didn't impress me.

Terence Eden is on Mastodon

@edent

File under "Falsehoods Programmers Believe About Families." pic.x.com/3vgiq0ursf

---

Let's take a look at some of the more baroque requirements in their terms & conditions.

Does your family have more than five members? Around 10% of babies born in the UK have three or more siblings. It also ignores families which want to include grandparents and in-laws in their group.

Or, "Sorry little Timmy, we can only add you to the Google family plan once your granny dies."

You're adopted, having been with a few foster families this year. You can't join the same Google family as your new brothers and sisters.

You split up from your partner - you can't join a new partner's family for a whole year.

In the UK, you can marry and/or start a family at 16. Are a young married couple really excluded from sharing their media?

The credit card requirement also seems odd given that you can purchase media using pre-pay cards with no age restriction.

Your adult child is about to be deployed overseas. Can they remain in the family so you can buy them games & films?

You live with your mum & step-dad. Your bio-dad wants to add you to his Google family. What do you do?

Why does Google have such a narrow conception of what a "family" is?

Falsehoods programmers believe about the "falsehoods programmers believe" meme

A number of people on Twitter have taken the title literally. "This is about managers and lawyers!" they protest. But, of course!

Terence Eden is on Mastodon

@edent

Replying to @edentOMG! The number of people who think I *literally* blame programmers!

The "Falsehoods" meme is about exposing poor planning at *all* stages.

---

All of the "Falsehoods" memes - including the original - are a shorthand for "Constraints placed upon a complex system by a mixture of ignorance, apathy, business requirements, or other legal constraints".

That's not quite as snappy!

This is undoubtedly a licensing issue. Google have used "Family" as a consumer-friendly term for "Group of people who want to share media but may or may not be related." The nuclear ideal of Mom, Pop, Dick, Jane, and one miscellaneous doesn't really fit with many people's reality.

If I buy a physical DVD or videogame, I can lend it to as many people as I want - whether they're part of my family or not. The friction of dealing with moving physical items is really the only barrier to me lending out my entire collection to the whole world.

The digital world obliterates friction.

So we're stuck with a situation where a mega-corporation and their lawyers try to define the minimum viable family which will keep other lawyers happy and, hopefully, won't piss off too many real families.

How do you think they're doing?

Thoughts?

You can continue the conversation on this blog, or this Twitter thread, or on BoingBoing where Cory has written some excellent insder information on the decision making process which produced this.

Yesterday, I received this delightful email from the publishers Simon and Schuster.

NEW book by the "Prophet of Silicon Valley", Jaron Lanier, available for your review!

Dear Mr. Vogels,

I'm writing today to let you know of a new book from Simon & Schuster that I think your readers will be interested in — WHO OWNS THE FUTURE? By Jaron Lanier (available 5/7/13).

Jaron Lanier is the bestselling author of You Are Not a Gadget, the father of virtual reality, and one of the most influential thinkers of our time.

[Snip a load of marketing fluff]

I'd like to send you a copy of the book to review.

Well, let's ignore the fact that I am not "Mr Vogels". PR is a hard job and getting people's names right isn't a fundamental part of relationship building.

So, as the book sounded interesting, I asked to be emailed a copy.

I'm happy to send you a hardcover copy. If you might provide a mailing address, I'll put one in the mail today.

As regular readers know, last year I burned all my paper books and switched exclusively to ebooks. I related that fact, and mentioned that it would be much quicker than international shipping - surely a priority if you want advanced reviews.

Unfortunately, I'm not able to send ebooks as review copies at this time. Many thanks for your interest.

So, the "Prophet of Silicon Valley" wants his words stamped on to bits of dead tree, then air shipped at great expense across the Atlantic - as opposed to emailing me a few MB?

How very futuristic.

Don't Copy That Floppy!

I don't know if this is Simon and Schuster police on ebooks - or Jaron Lanier's. Given Jaron's stance on DRM and his publisher's ongoing DRM law-suit it could be either.

What I do know is this:

• Regional release dates are antithetical to the Internet. The UK edition of "Who Owns The Future" went on sale in the UK on 7th March - it goes on sale on May 7 in the US/Canada and May 22 in Australia/NZ
• 1 minute of Googling found an eBook copy, without DRM, nestling on a Russian server. Along with the usual polarity of Magnet links.

It just goes to show, even the most influential futurist, and his publishers, are still struggling with the reality of the Internet and the ongoing eBook revolution.

The book certainly sound very interesting. To quote from the preface:

The very action of reading makes you the hero of the story I am telling. Maybe you bought, or stole, a physical copy, paid to read this on your tablet, or pirated a digital copy off a share site. Whatever the prequel, here you are, living precisely the circumstances described in this book.

If you paid to read this, thank you! This book is a result of living my life as I do, which I hope provides value to you. The hope of this book is that someday we’ll all have more ways to grow wealth as a side effect of living our lives creatively and intelligently, with an eye to doing things of use to others.

If you paid to read, then there has been a one-way transaction, in which you transferred money to someone else.

If you got it for free, there has been a no-way transaction, and any value traded will be off the books, recorded not in any ledger but rather in the informal value systems of reputation, karma, or other wispy forms of barter. That doesn’t mean nothing has happened. Maybe you’ll get some positive strokes over a social network because of what you say about the book. That sort of activity might benefit us both. But it’s a kind of benefit that is unreliable and perishable.

"Who Owns The Future?" is available from all good book stores, as both a hardback or eBook. Jaron Lanier would probably prefer you buy it, but is probably equally wary of your personal details ending up in a mega data warehouse.

As requested, I recorded all the sessions I could. Here are my notes, and the audio from each session I attended.

I've based some of the information from the excellent LibraryCampLondon Wiki, including the session proposal page.

Hidden Collections

Katie Birkwood lead a fascinating session. It turns out that there are hundreds of thousands of books hidden away in libraries. They're not in the records because of a lack of decent cataloguers. Which means people can't find them. Which means they're rarely used. Which means penny pinching bureaucrats want to "dispose" of them - homely or otherwise.

One suggestion I made was to use "negative metrics". Look at your search results - what common queries receive no results? Those are the books you need to prioritise for cataloguing.

Audio

Tweets

https://twitter.com/NazlinBhimani/status/307811824257077248

Chris B.

@libchris

crowdsourcing - how many have message on catalogue that invites contributions? (COPAC prominently says if an error let us know) #libcampldn

---

Overdrive DRM is Broken (and that's ok)

I gave a demonstration on just how ridiculously easy it is to circumvent the ePub DRM system used by OverDrive - the system used for ebook lending in many libraries.

I used the metaphor of a locked box of doughnuts. I handed the box to a willing victim and told her that she may only unlock the box if she promised only to eat one doughnut and not to share any with her friends. I then gave her the key. Shortly afterwards the group had scoffed the lot! That's what makes DRM impossible; you give the lock and the key to the same person and expect them never to behave dishonourably.

I then facilitated a discussion around whether DRM was ever justified, whether libraries should boycott OverDrive, and if it was legal for a librarian to tell a patron how to circumvent artificial restrictions.

Audio

Tweets

I'm happy to say the session was generally well received.

shedsue

@shedsue

Most exciting session of the day IMO Hacking Overdrive #libcampldn NOW! With @edent

---

LibraryCamp

@LibraryCamp

Replying to @edent@edent on hols with kindle. Ebooks in libraries. Prob 1) two weeks loan - why - this is a digital file! WTF #libcampldn #ebooks

---

Lloyd Davis

@LloydDavis

Much anxious excitement in the group as @edent showed how straightforward DRM is to break on epubs #libcampldn

---

Claire Back

@calire

eBook session one of the best I’ve been to at a Library Camp. @edent and not just because of the doughnuts. #LibCampLDN

---

Facilitating the "Coming of Age" of Open Source Software in Libraries

Primarily this was a discussion about the Open Library Environment. An innovative foundation to open source many parts of the library environment.

We also discussed how to use other open source products - like FireFox on Internet connected PC.

Annoyingly, one of the participants asked for the session not to be recorded - which I have no problem with, but the guy didn't even speak during the discussion. What's that all about?

CodeClub

Greedily, I ran a second session after being asked to talk about my experiences teaching kids to code using Scratch.

I talked about the CodeClub.org.uk website and resource - telling the librarians how they could get involved.

I even showed off some of the Scratch games I had written. Hopefully it will encourage more libraries to run coding sessions for kids.

Audio

Comics

Try as I might, I just can't get in to graphic novels. I used to love the Beano as a kid, but most modern comics leave me cold.

It was, therefore, fascinating to hear about the rich tapestry of comics within libraries. How they're often used as gateways for reluctant readers. How inexperienced librarians often misfile them. And how explicit manga is not always suitable for young readers.

My resolution - to try to find at least one comic that I really enjoy. I've been recommended Maus (the story of the holocaust told from the perspective of some mice) and Kill Shakespeare (a frankly bonkers epic tale about all of Shakespeare's characters).

Audio

Overall

A really worthwhile and invigorating experience. Senate House Library is a delightful venue - although the layout of the rooms made the sessions a little noisy at times.

I enjoyed coming to an unconference which was so diverse, and had almost nothing to do with my day-to-day life. I'd recommend that every BarCamp attendee participates in an event which is totally outside of their normal life.

This is a long and rambling blog post which asks two questions:

1. Given that we can only read one book at a time - can we set up a "Kindle Stock Exchange" for our unused media?
2. If so, what effect would this have on content producers and sellers who rely on artificial scarcity?

This has been mulling over in my head for some time.

Image from FreeFoto.com

Transactional Friction

One of the great advantages of the digital era is the reduction of transactional friction.

The time, effort and money it takes to perform a transaction in the online world is radically reduced from that of the physical world. Sending a letter to the other side of the world used to take weeks - and involved ships, planes, and physical delivery. Now a click of the mouse sends it anywhere in an instant.

Neal Stephenson's novels in the Baroque Cycle document the rise of the bank note. It used to be common practice for wealthy people to lug their gold with them when they went on journeys. This was, as you can imagine, a highly inefficient practice - it was slow, dangerous, and expensive. The bank note reduced the friction of having to physically transport gold.

The efficiency of being able to settle large debts without the need to physically move goods was a revolution which introduced the free movement of capital, and dramatically altered the nature of debt and lending. It shaped the modern world.

The digital world can also free us from the needs of the physical. While many of us already have some goods that only exist as 1s and 0s on disk - some pioneers have already shifted as many of their possessions as possible away from the physical world. They've turned their heavy, impractical, and inefficient gold into a light, practical and frictionless bank note. A lifetime of books, films and music - stored on a lightweight thumb drive. Or floating weightless on the Internet.

That's why, last year, I decided to burn all my books.

Making Money

Here's a picture of my book shelf. At any moment, I can only read one book at a time, watch one DVD, and listen to one CD.

Would it make sense to sell all the books and DVDs I owned except for the one I was currently using? When it comes to physical books - no.

It would be so inefficient. It takes too much time, money, and effort to sell a single book than the value I would recoup from doing so. Sticking a book on eBay, postage, packing, dealing with refunds etc. is a lot of effort for relatively small gain.

The digital world changes all of that. While the ability to instantly consume is exciting, what intrigues me is the ability to take my goods to market - and the consequences that would have on capitalism.

What would happen when - rather than buying books instantly - I can offer them for sale instantly?

Selling My Bookshelf

Imagine if you could easily rent out your DVDs - like a mini-BlockBuster. They're sat unused on the shelf all night, ignored while you're at work during the day, left unplayed when you go shopping.

The hyper-efficient online world makes this possible.

Just like taking a loan secured on my assets (car, house), I can release equity from my digital goods by selling them when I'm not using them. Which is most of the time.

99% of the time, your DVD collection isn't in use. You may share them with your family, friends, and co-workers - why wouldn't you share them with everyone?

There are some very good reasons why not in the physical world - access, lack of selection in your collection, wear and tear - but in the frictionless online marketplace, most of these concerns disappear.

Just as on eBay, your single DVD sits on the same shelf as someone selling thousands of DVDs. The buyer probably doesn't care about range of selection or the providence of a seller - she just wants the item she desires.

With digital copies there are no issues of wear and tear and an online copy of a work is instantly accessible.

A second hand digital work is identical in quality to a factory fresh copy. My ebook edition of Little Brother is bit-for-bit identical to one bought anywhere. In a perfect market, it would fetch the same price as one purchased from a retailer. Or, more likely, the retailer would reduce the price to the lowest available on the market.

(This leads us down an interesting path. The whole notion of "second hand" or "used" goods disappears when we're talking about digital copies. But that's for another post.)

Real Time Trading

We can even go a step further. Stock trading used to be conducted by sweaty men shouting in arenas. There are now electronic agents on the the stock exchange which process millions of trades per second. Every millisecond the computers execute trades based on real-time information.

I've said previously that you can only read one book at a time. Your eBook reader will only display one page at a time - so why not sell the pages that you aren't reading? In the physical world, perhaps two people can read a book at the same time (if they're close and not overly annoyed with the other's reading speed). In the digital world, you could have one reader for every page in the book.

The second you click "Next Page" your eBook reader could instantly find the cheapest price for the page you want to read - and sell the pages you're not reading for the best price.

A DVD could be separated into scenes or - to push it to the extreme - individual frames. If I sell my movies when I'm not watching them - why shouldn't I lend or sell them while I am watching them?

The Effect

Moving from transporting gold to transporting bank notes to digital trading has radically changed the nature of money and banking. The free movement of capital completely disrupted the way the world works. The free market dominates economic thought. But it is only a free market between giant players. Only corporations can afford to play capitalism's game.

Will the free market be brought down to individuals? Will a grandmother in her living room be able to sell her digital goods as easily and freely as Warner Brothers?

What does the media publishing landscape look like when every consumer can change their bookshelf into a second hand store? Why would anyone buy a copy of a work from Amazon when they could borrow from their neighbour - even if that neighbour lived on the other side of the world?

Would we see multiple Bourse to facilitate the exchange and a willingness for people to sweat their assets?

Perhaps it will move us to "subscription" models for mixed media - taking a £50 per month subscription to books in the same way many of us do for television channels.

What does it even mean to sell goods when they can be replicated instantly at zero cost?

I don't know - but I think it will be exciting to witness.

There Is An Analogue Hole In Your Mind

Applying analogue thinking to the digital world leads us down some interesting rabbit holes. As a thought-experiment, it's incredible to imagine that we could end up in a crazy world where we pay-per-word, where the right to read is predicated on software stock brokers trying to extract every micro-cent of value out of copyrighted works.

We need to break out of our old ways of thinking. The zero-friction economy is a paradigm shift (in the sense of Kuhn). We need to adapt our modes of thinking and our economies to this brave new world.

Looking at the prices of eink devices at wholesalers, this looks broadly accurate. They do seem to be selling at around wholesale cost - customers also get Amazon's fabulous support, free software updates, and high quality manufacturing.

Yet there is a curious anomaly. Why aren't Amazon selling ePub books?

Terminology

A quick diversion into the terminology used in this article.

• eReader - the physical hardware. Kindle, Kobo, nook, etc.
• eBook - the electronic file containing the words & pictures. ePub, Mobi, PDF, etc.

Background

There are, broadly speaking, two main formats for ebook - ePub and MobiPocket. Think of them like the difference between 8-Tracks and cassette tapes - they both hold music, but play on different system.

ePub works on just about every eReader on the planet - with the notable exception of the Kindle.

MobiPocket (or Mobi, for short) only works on the Kindle⁰.

Wikipedia has a fairly comprehensive comparison of which device can handle which format.

So, we have a problem. The books you buy from Amazon can't be read on your Sony, Kobo, Nook, or Generic eReader. Well, they can, but you have to remove the DRM, covert the book to the ePub, and hope that everything works ok.

What a pain in the arse.

What Would Happen If...

Now, I'm not suggesting that the Kindle should be able to read ePub books. Obviously, it's technically capable of doing so - but it would mean that Amazon customers could compare prices with other retailers and start to leave the Amazon ecosystem.

What I'm suggesting is that Amazon should say "Buy this ebook for your Kobo" and deliver an ePub to those poor, unfortunate souls who haven't been blessed with a Kindle.

There are lots of statistics regarding eReader share. Some suggest that Amazon have a ~47% share of the eReader market in the US whereas the Kobo eReader has a 46% share in Canada, and 50% share in France.

Let's say that the Kindle has a worldwide share of 50%. Amazon has two options:

1. It can aggressively pursue that market share by producing more innovative, cheaper hardware, and hope to convert users to the Amazon flock
2. It can accept that some people don't want its hardware and start selling books directly to those users

Amazon claims that it makes more money from eBook sales than hardware. So why doesn't it expand its market to the 50% of eReaders which are currently not served by its store?

At the moment, customers with Kobo, nook, and other eReaders can compare prices across a number of eBook stores. What would happen if they could add Amazon to the list of shops they could compare with?

The documentation for MakeMKV is available. It's a bit sparse. It also doesn't have many of the features of the GUI.

For example, the GUI allows you to select which video, audio, and subtitles, you want to copy. So if you only want the original German director's commentary, with the Swedish subtitles from the main feature, you can do that. The command line is an "all or nothing affair".

Install The Software

We're going to be using MakeMKV. The latest version of the Linux software can be found on their forum.

First, download both pieces of software.

wget http://www.makemkv.com/download/makemkv-bin-1.7.6.tar.gz
wget http://www.makemkv.com/download/makemkv-oss-1.7.6.tar.gz

Then, extract the files.

tar -zxvf makemkv-bin-1.7.6.tar.gz
tar -zxvf makemkv-oss-1.7.6.tar.gz

Get the tools you need to build the software

sudo apt-get install build-essential libc6-dev libssl-dev libexpat1-dev libgl1-mesa-dev libqt4-dev

NOTE: You need GCC version 4.6 or higher. If you're unsure which version you have, type:

gcc --version

If you need a more recent version, follow these GCC upgrade instructions on StackOverflow.

To build the software, first compile the Open Source part

cd makemkv-oss-1.7.6
make -f makefile.linux
sudo make -f makefile.linux install

Then install the binary part (you will have to agree to a licence).

cd ../makemkv-bin-1.7.6
make -f makefile.linux
sudo make -f makefile.linux install

All should be installed without error!

Running for the First Time

Stick a DVD in your drive and run the following command

makemkvcon -r info

You should get a bunch of information about the drive and the program.

To see information about the disc, run

makemkvcon -r info disc:0

Assuming that your DVD drive is 0.

How To Rip

This command rips all the movies on the disc which are longer than 3600 seconds (60 minutes). This means that it should skip any special features etc.

You will need to create the folder you want to rip to first. The filename will be "title00.mkv" so make sure you're ripping to a unique folder each time.

makemkvcon --minlength=3600 -r --decrypt --directio=true mkv disc:0 all /DVDs/MovieName/ ; eject -r

The " ; eject -r" tells the system to eject the disc once ripped.

What Can't You Do?

MakeMKVcon is missing the following functionality (hopefully being fixed by its creator)

• Can't select a specific title.
• Can't extract specific audio files.
• Can't specify subtitles.
• Can't create folder.
• Can't set filename.

Well, I say a "worrying flaw" - I mean an error. All books contain errata - I think that's a given - but outside of academia, Jasper Fforde is the only author I know who offers upgrades to his books.

Here's a sample from the original Thursday Next "patch"

5: Using a fine black pen make the following corrections:

6: Go to page 32 and replace 'Stella' with 'Steller's - this is the correct spelling. The large slow-moving-manatee-type-mermaid-legend creature was named by Georg Wilhelm Steller, the German naturalist.

What's The Error?

The book contains a number of charming illustrations - the final illustration is meant to be (NO SPOILERS) about a wiped out clown army. Instead, it's a repeat of an earlier illustration of (NO SPOILERS) mimes encircling a car.

At the time of writing, there's no upgrade listed for the latest book - although there are a range of fabulous special features. I've dropped Mr Fforde an email alerting him to this devastating turn of events.

Yes, yes, it's fairly minor in the grand scheme of things.

Expecting More

Of course, one could argue that traditional books don't get upgrades - so why should this be a problem for ebooks?

• I can take my physical book back to the shop and get a replacement. Or even send it back to the publisher. With eBooks, this is virtually impossible - not least because of the DRM issues involved in revoking a book.
• If a book contains a serious error, I can print out an errata sheet. The DRM on eBooks prevents me from altering their contents.
• We should expect more. This is a new medium - we should expect more than simply plain text in a DRM layer.

Yes, it all comes down to DRM - or, as Jasper Fforde puts it...

The DRM was the Dark Reading Matter – the unseeable part of the BookWorld.

and

DRM’s existence remained theoretical, at best.

I don't know if I'm reading too much in to Fforde's work - but he doesn't strike me as the sort of author to use an acronym without a thorough understanding of what it means. I have no shame in saying that I removed the DRM on the ePub I purchased in order to read it on my Kindle (which does not support Adobe's DRM scheme).

I don't know how I expect an eBook upgrade to work - with or without DRM. I don't want an author, book seller, or publisher to randomly change the book I'm reading - that's a little too similar to Amazon deleting 1984 from its Kindles.

I know I don't want to buy an entirely new copy - just because some punctuation has been fixed.

Should I be able to download a diff and let my eReader decide which version of a book I want to see?

Should I be able to get an update free? Should it cost?

So many questions and so few answers. If you've got any thoughts on the matter - please let me know.

eBooks and eAudiobooks from Surrey Library (click to embiggen)

"Bet the site doesn't work on my phone," I thought.

http://ebooks.surreycc.gov.uk/

I was wrong!

Overall, this is a brilliant new service. A great initiative to get people reading more books and improving library services.  Take a look at ebooks.surreycc.gov.uk

There's just one problem...

DRM

Digital Restrictions Management - as the name implies - prevents you from performing certain actions with a file.  As it happens, I don't disagree with the notion that a rented work should have restrictions on its use.  These books are given to me for free - I don't own them.  DRM serves to enforce the tacit social construct that - if I borrow a book from you - I'm expected not to lend it on and return it after a reasonable period.  That's fine with me.

What I do disagree with is artificial scarcity.  In the "real" world, if you only have 3 copies of a book, you can only lend out 3 copies.  In the digital world, if you have one item, you can copy it an infinite number of times.  Pick a file, hit CTRL+C then CTRL+V - instant copy.  And you can do it as many times as you like.

In a physical library, only one person can borrow a book at a time.  In a digital library, it makes no sense to say there is only one copy.

The title can be lent out to as many people who want it.

Some books are - bizarrely - more available than others.  Why can some books have 7 copies and others only 1?

Which leads us in to the incredible situation of being on a waiting list for a digital file!

This artificial scarcity also has an impact on the number of books you can borrow.  If I'm going on a two week holiday, why shouldn't I be allowed to borrow as many books as I like?  Taking those books from a physical library would stop other patrons enjoying the works - in a digital library anyone can borrow any work simultaneously.

Finally, we have the issue of copying.  One of the reasons I use libraries is for research.  In fairness, the DRM does allow for limited copying.  But different books have different allowances despite - presumably - being covered by the same copyright regime.

This is madness.  Why take something as amazing as the digital revolution and try to drag it down to the mundane level of the analogue world?  It's like inventing the motorcar and insisting that drivers carry a bale of hay, replacement horseshoes and travel no faster that 50mph.

Compatibility

The side effect of this wasteful DRM is a lack compatibility with popular products.

The MP3 and WMA audiobooks require the OverDrive Media Console.  ODMC is compatible with PC and Mac - but not Linux.  It works on Android, BlackBerry, iPhone and Windows mobile - but not the billions of Symbian or J2ME handsets out there.

The DRM on the eBooks requires Adobe's proprietary Digital Editions software.  It's not available on Linux and - judging by the comments on my blog - it doesn't work very well on Windows or Mac.  It's also restricted to a few ebook readers - excluding Amazon's Kindle.

So, due to DRM, we're left with a library service which...

• Artificially restricts how many books are available.
• Forces people to buy a particular brand of eBook reader or mobile phone.
• Requires the use of Windows or Mac.
• Prevents the early return of some media (you can't "return" some books early. Why?)
• Stops the quoting of passages for future use.

Overall - what Surrey Libraries have done is take an amazing service and cripple it.  The potential is there - they just need their mindset moved to the 21st century.