Investigating: Opening the PDF in LibreOffice Draw allowed me to see that the watermark was a separate image floating above the others.

Manual Solution: Hit page down, select image, delete, repeat 500 times. BORING!

Further Investigating: Using pdftk, it's possible to decompress a PDF. That makes it easier to look through manually.

pdftk input.pdf output output.pdf uncompress

Hey presto! A PDF you can open in a text editor! Deep joy!

Searching: On a hunch, I searched for "watermark" and found several lines like this:

<<
/Length 548
>>
stream
/Figure <</MCID 0 >>BDC q 0 0 477 733.464 re W n q /GS0 gs 479.2799893 0 0 735.5999836 -1.0800002 -1.0559941 cm /Im0 Do Q EMC 
/Figure <</MCID 1 >>BDC Q q 28.333 300.661 420.334 126.141 re W n q /GS0 gs 420.3339603 0 0 126.1418879 28.3330078 300.6610601 cm /Im1 Do Q EMC
/Figure <</MCID 2 >>BDC Q q 16.106 0 444.787 215.464 re W n q /GS0 gs 444.7874274 0 0 216.5921386 16.1062775 -1.1281493 cm /Im2 Do Q EMC
/Artifact <</Subtype /Watermark /Type /Pagination >>BDC Q q 0.7361145 0 0 0.7361145 113.3616638 240.8575745 cm /GS1 gs /Fm0 Do Q EMC
endstream
endobj

Those are Marked Content Blocks. In theory you can just chop out the line with /Subtype /Watermark but each block has a /length variable - so you'd also need to adjust that to account for what you've changed - otherwise the layout goes all screwy.

That led me to PyMuPDF which claimed to solve the problem. But running that code only removed some of the watermarks. It got stuck on an infinite loop on certain pages.

So, now that I had more detailed knowledge, I managed to get an LLM to construct something which mostly seems to work.

Does it work with every PDF? I don't know. Does it contain subtle implementation bugs? Probably. Is there an easier way to do this? Not that I can find.

import re
import pymupdf

# Open the PDF
doc = pymupdf.open("output.pdf")

# Regex of the watermarks
pattern = re.compile(
    rb"/Artifact\s*<<[^>]*?/Subtype\s*/Watermark[^>]*?>>BDC.*?EMC",
    re.DOTALL
)

# Loop through the PDF's pages
for page_num, page in enumerate(doc, start=1):
    print(f"Processing page {page_num}")
    xrefs = page.get_contents()
    for xref in xrefs:
        cont = doc.xref_stream(xref)
        new_cont, n = pattern.subn(b"", cont)
        if n > 0:
            print(f"  Removed {n} watermark block(s)")
            doc.update_stream(xref, new_cont)

doc.save("no-watermarks.pdf")

One of the (many) problems with Vibe Coding is that trying to get a LLM to spit out something useful depends massively on how well you know the subject area. I'm proud to say I know vanishingly little about the baroque PDF specification - which meant that most of my attempts to use various "AI" tools consisted of me saying "No, that doesn't work" and the accurs'd machine saying back "Golly-gee! You're right! Let me fix that!" and then breaking something else.

I'm not sure this is the future we wanted, but it looks like the future we've got.

The main point, I agree with. Citing a whole paper is a lossy process. Saying "Smith, J (1963) Practical Time Travel, Gallifrey Press" is basically fine but it doesn't tell you where in the publication the bit is you're citing.

Xiao & Ballou contend that the answer is to add a page number to every citation. I think that's a simple solution - but almost always the wrong way to solve the problem.

Firstly, page numbers aren't stable. If you've got the large-print version of a paper, it will have a different page numbers than the regular print edition. Paperbacks and hardbacks have different numbers. The paper copy might be formatted differently from the digital copy.

Secondly, most modern documents simply don't have pages. Anything published as HTML / ePub won't have a page number. Page numbers are a skeuomorph in those contexts.

Finally, even if there is a stable page number - is that precise enough? A page may have several hundred words in multiple sections. Page numbers aren't granular.

I've ranted before about I consider Quoting Page Numbers from eBooks is Considered Harmful.

So what's the solution?

URl fragments!

A properly formatted document should be accessible by sub-section. For example:

example.com/papers/time-travel.html#2.3.9-Eye-of-Harmony

That takes us to the paper we want, and straight to the anchor heading we're discussing. This also works with the obsolete PDF format.

Now, not all papers have the correct markup to do this. That's OK! We can cheat using Text Fragments. That allows us to link to the first instance of a specific string of text in a document. For example:

example.com/papers/time-travel.html#:~:text=Reverse the polarity of the neutron flow

I genuinely think both of those solutions make more sense than hoping that your reader has a copy of a paper in exactly the same format as you do.

One of the things I found most frustrating during my MSc was how... old fashioned academia was. PDFs with two column layouts, unreadable fonts, paywalled knowledge trapped in moribund formats. Let's try to drag it into the late 20th Century, eh?

Whole industries grew up around them - one of my first jobs was programming binary loadlifters repairing computerised cheque-readers - they're an example of a technology which "just works".

Which, of course, is nonsense. They're fragile, easy to forge, have little protection, and are cumbersome to use for all parties - they're just generally terrible. At this stage of a legacy technology, we generally have two choices - replace or double-down.

Most of the modern world has replaced cheques with electronic banking. Whether its the innovative mobile services of Africa's M-Pesa or the somewhat stodgy British Direct Debits - money flows electronically. Low latency, low cost, low effort.

America, however, has decided to pile on successive work-arounds for the inconvenience. This cool app lets you take a photo of a check and have it magically deposit in your account! (After five days.)

What a pile of nonsense. Shuffling paper around from one end of the country to the other, at high latency, high cost, and high effort. It's just bollocks, isn't it? No matter how many iPhone-based band-aids you stick on.

So we come to the PDF.

The Chilcot Report into the Iraq War has just been released. As PDF. Fifty separate PDFs.

Terence Eden is on Mastodon

@edent

As predicted (read the thread) #Chilcot is PDF only. Totally unsuitable for linking, searching, reusing. x.com/edent/status/7…

Terence Eden is on Mastodon

@edent

PDF needs to be banned as a Government publishing format. x.com/robertsharp59/…

---

---

Terence Eden is on Mastodon

@edent

Replying to @edentAnd much of the #Chilcot evidence is PDFs of scans of photocopies of printouts!
Impossible to search & critique. pic.x.com/3xqjtjnfdq

---

This is despite repeated requests that it should be available in a more suitable format.

Let me be clear. Releasing a PDF is a hostile act. It says you care more about your own convenience than your users'.

As described in The New Statesman, this is an intersection of politics, open standards, and technology.

Let's take a quick look at PDF's sins.

Hard to reflow

If you're reading a PDF on a digital device - because who prints out to paper any more? - you'll probably want to zoom in on the text. Here's what happens with the Chilcot Report if you ask Android to reflow the text:

Inaccessible.

Supposedly accessible PDFs fail in horrendous ways - even when trying to do something as simple as copying text from them.

Terence Eden is on Mastodon

@edent

Replying to @jonbmetz@jonbmetz @robertsharp59 @hfiguiere I'll stick to plain HTML, thanks. pic.x.com/iyvgie2aba

---

There are many technique you can use to make a PDF accessible - just like with HTML - but it would appear that the Chilcot Report doesn't take advantage of any of these.

Difficult to extract data.

Quick - find every reference in these PDFs to "Camp David". You can't. Quick - disambiguate between Camp David, Sir David Manning, David Brummell, Dr David Kelly. You can't. There is no semantic markup.

Of course, some people see this as an advantage.

RT[redacted]: Crushing news that people will need to read shit and not just hop from keyword to keyword https://twitter.com/edent/status/750652342853365760 — Matilda (@tillywrites) July 6, 2016

Huge attack surface

PDF is a horrifically complicated standard. So complex that it leads to all sorts of terrifying ways to abuse the reader.

(That's not to say that HTML5 isn't also complex, of course.)

Bloated file size.

Combined, those PDFs add up to around 40MB. Great if you're on a fast connection - not great if you're paying per MB on your phone.

Converted to a plain text format like markdown, it's a mere 370 KB. That's a full two orders of magnitude smaller.

Hard to link to specific parts

PDF does make it possible to link to specific pages - but pages are a skeuomorph. We want to be able to link to specific paragraphs.

While the reports contain some external links, there are no internal links between the documents - making navigation and cross referencing even harder than necessary.

And So It Goes

Other people have written about PDF's failings with far more eloquence than me. This isn't a new rant - Jakob Nielsen was criticising PDF back in 2001.

I truly believe that the Internet needs to treat the PDF as harshly as it treated Flash. We should be embarrassed that such legacy technologies have been allowed to create a stranglehold on our creativity. They are stifling our democracy by trapping vital information in a digital tar pit.

We must drive PDF out - cast it to the winds - make it as impolite to use as auto-playing MIDI on a website.

You wouldn't accept being paid by paper cheque - why should you accept receiving data by PDF?

Tom Steinberg

@steiny

When I'm king public servants who release PDF-only reports will be auto-fired by a filetype detecting script connected to the HR system

---

If you'd like to help convert the Chilcot Report to a more open, accessible, and semantic format please get involved with official-inquiries.com

A little backstory... The Europeana Carare Project has some gorgeous 3D scans of ancient monuments and heritage sites. The only problem is that the 3D files are trapped inside PDFs. Yup - did you know the PDF spec supported 3D models? Me neither.

The only way to view these is with the Official Adobe Viewer (install it on Linux following these instructions).

This is what it looks like:

Nifty, right?

So, how do we extract that 3D file?

1. Download and install the Pythong PDF Parser
2. Search for the 3D object:
   ./pdf-parser.py -t /3D input.pdf
3. You'll get a response like this:
   

   obj 48 0
    Type: /3D
    Referencing: 58 0 R, 59 0 R
    Contains stream
   
     <<
       /AN 58 0 R
       /Length 6066066
       /Subtype /PRC
       /Type /3D
       /VA 59 0 R
     >>
   

4. The Object ID (in this case 48) is what we need to extract.
5. ./pdf-parser.py -o 48 -d output.u3d input.pdf

You can download the U3D file.

Unfortunately, nothing opens it. I've tried Meshlab, Blender, FreeCAD, i3D Converter, online service Babel3D, and even Unity 3D on a Windows box.

Nothing! Not a single application would recognise it - let alone convert it.

There is a U3D library on GitHub - but I'm not skilled enough to integrate it into anything.

I've found an Android app which claims to convert U3D files - but it has no downloads, no reviews, and costs £6!

So, gentle reader, if you know of a way to extract and view 3D models from PDFs, please let me know!

There's just one small niggle I have with it. It is released as a PDF.

PDF is a terrible format for ebooks. Releasing a PDF ebook is the publishing equivalent of releasing an iPhone app. 99% of the time an iPhone app is overkill for what a company is trying to achieve and more often than not completely ignores the target market.

This is something that Tomi has mentioned time and time again on his blog.

Tomi would be better off releasing his book as a mobile website if he wants lots of people to read it on his phone!

I am being slightly unfair on Tomi - but I know that he can take it as well as he dishes it out. He's not a publisher and shouldn't be expected to know every single ebook format out there. But his publishers should.

He and I bother decry the "iPhone first!" mentality - quite rightly. So I am going to educate you on why "PDF First!" doesn't cut the mustard in an ebook world.

Text Reflowing

This is the most important reason. PDFs place hard line breaks at the edge of the page. But what do we mean by "page" on an ebook? Because we can resize text and adjust our margins, the idea of a break at the edge is outdated.

Take this paragraph.

The biggest market for MMS is clearly Asia. As far back as 2008, 48% of Asian mobile phone subscribers were using MMS according to a TNS survey across 29 countries in the Asia-Pacific region.

On a phone or an ereader, the text is likely to render like this:

The biggest market for MMS is clearly Asia. As far back as 2008, 48% of Asian mobile phone subscribers were using MMS according to a TNS survey across 29 countries in the Asia-Pacific region.

The ereader wraps the first line, then it encounters the line break and starts a new line. It does this because it has no way of knowing whether a line break is there because it's the end of the paragraph or the end of the page.

Some ereaders do a better job than others - but even the Kindle isn't immune.

Worse, some phones will not reflow the text at all - meaning you have to scroll right to read a single line, then left all the way, then down all the way. Not the best reading experience in the world. This is how the Kindle - the best eReader on the planet - handles Tomi's PDF.

Fit To Screen View

You can click on the image to see the full resolution version - it's just about readable, even with those large margins (not needed on an ebook!).

What happens if we zoom in?

Zoomed in so it's readable

Much more readable - but we have to scroll left and right to read.

That's the primary reason why PDF sucks. What are the others?

File Size

Let's take a look at a popular free book - Jane Austen's Pride and Prejudice.

The ePub and Kindle formats are zipped - so they are really small. The PDF is twice the size of even the uncompressed plain text version.

Not all of us are lucky enough to live in a world of unlimited download allowances - or even in a land of 3G coverage - and even Kindles have a limit to the amount of memory they have. Making your files as small as possible is a courtesy to your readers.

Security

There are just too many flaws in the PDF reader to make it viable to be installed on handsets or computers. As long as Adobe are continually offering patches, most people would be wise to avoid PDF unless they absolutely need to.

Widespread Adoption

I'll admit, I don't know how many phones have a PDF reader - let alone have a good PDF reader. Whatever the number, I guarantee it to be lower than the number of phones with a web browser.

From Tomi's excellent post on The Year In Stats

100% can do SMS - this is 4.25 Billion phone handsets that can send and receive messages 97% have at least a basic browser (including WAP, not necessarily color) 95% have a color screen - this is 4 Billion, more than twice the number of TV sets and 3x number of PCs 92% are data-capable (at least 2.5G ie GPRS or basic CDMA 2000) 85% support MMS - this is 3.6 Billion, more than twice number of TV sets and 2x active users of eMail 81% are cameraphones - this is 3.4 Billion cameras 76% have a full brower ie HTML type of browser (compare with 17% which is total number of smartphones) 62% have a media player 61% support apps using Java or Brew (compare with 17% which is total number of smartphones) 51% have a memory card slot 35% are 3G phones (not nearly all are on 3G networks) 21% support WiFi 17% are smartphones 12% are second-hand phones (mostly in emerging world countries, but also with younger kids)

No mention of how many phones support PDF! While some modern smartphones like the N95 have a native PDF reader, it's often of low quality and doesn't make for a pleasant reading experience.

Conclusion

I feel really guilty for beating up on Tomi. His book looks wonderful and I'm sure I'll enjoy reading it. I thank him, quite sincerely, for all the advice he dispenses and sanity he injects into the mobile landscape.

He has also been very generous with the terms and conditions of the book. While not explicitly Creative Commons, he does allow the book to be freely copied and quoted - even printed out!

I should also note that in his introduction he says:

Furthermore, if you want a smaller, condensed version optimized for smartphone reading, there is a paid mBook/smartphone version for your iPhone, Blackberry, or whatever pocketable device you might want. The mBook edition also will not be encumbered with the advertising, and will be made in a file size significantly smaller than the free edition, to help those who save the mBook on a smartphone that may have limited storage space. The mBook edition has the illustrations in color to help see some graphics more clearly on the small screen.

In fairness, this book is still in beta - I am confident that Tomi will release a final version which will be accessible to all.

I know he wouldn't give a publisher a free ride if they released an iPhone app where a mobile website would do. I respect him too much to stay silent when I see him releasing a PDF where a more suitable eBook format makes more sense!

I hope he accepts my criticism in the spirit in which it is intended.