Security-Scoped NSURL bookmarks and Safari’s webarchive files

It was a bug I’ve been carrying along for quite some time in Yoink. But I finally found the culprit: I’m looking at you, OS X sandbox.

Webarchive opened after bug occurredA webarchive created with Safari, after a security-scoped NSURL bookmark was created for it.

The Bug and its Detection

As it happens (sadly), not I discovered the bug, but a customer and user of my app Yoink encountered it. The reason being, I rarely handle .webarchive files (if at all) – webarchive files are created when saving a website in Safari, for example – but one of Yoink’s users, Christoph, appears to have to deal with them regularly.

The bug itself is described fairly easily. You have a .webarchive file you’d like to move using Yoink. So you drag it onto the app and then move it from Yoink to the actual destination. Business as usual.
Only that now, instead of opening the webarchive in your standard browser when double-clicked, above’s warning is shown.

There’s two baffling things about this warning:

  1. It looks like you’re trying to launch an application instead of just a file (“from an unidentified developer”)
  2. The creator of the file seems to have changed. Instead of Safari, it now says “BugReport-WebArchivesAndNSURLBookmarks” (the app I submitted to Apple to demonstrate this issue)

Highly concerning, to say the least. To some people, it might even look like something fishy is going on.

Hunting Down the Bug

Seeing as .webarchive files are binary property lists (thank you, Michael Tsai (@mjtsai on twitter) for the correction), I tried other property list files (for example, .plist files), but none exhibited the same behavior.

Reproducing it was fairly easy. There’s one pre-requisite for it to appear: that in System Preferences -> Security & Privacy -> Allow apps downloaded from, either ‘Mac App Store’ or ‘Mac App Store and identified developers’ be selected.
Otherwise, Gatekeeper isn’t active and doesn’t react to the issue.

System Preferences - Security and Privacy Settings

At first I thought it happened when moving the file out of Yoink, since that’s where Yoink actually does something to a file – it moves it from one location to another – who knows what goes on behind the scenes there.

Going through the move-code line by line, commenting out stuff I thought could cause this, made no difference what so ever. Create a new webarchive, move it using Yoink, and get the warning again. Rinse and repeat.
With stuff like this, I get annoyed easily, so my patience usually goes out the window after a couple of alterations to the code.

It was only after the billionth time that I considered it might happen when a file was added to Yoink, not moved from Yoink.

Going through the same process as before, I went through the code line by line, trying different things.

With a recent update to Yoink, the app knows when a file is renamed or deleted in Finder, using GCD (Grand Central Dispatch) and its dispatch sources.
I believed the issue lay there, but after commenting out all of that code as well, it became clear it wasn’t the culprit.

In a fit of anger and a severe feeling of incompetence, I randomly picked at the code (I know, highly professional. But I’m using version control – all is well, right?).

To my surprise, when I removed the code responsible for saving Yoinks files over relaunches of the app, the issue went away.

Security-Scoped NSURL Bookmarks

For a sandbox’ed app to keep a reference to a file added by the user beyond relaunches, it has to use what is called a security-scoped NSURL bookmark.

Sandbox entitlements necessary for security-scoped bookmarks

They can be used if the corresponding entitlement (see above) is added to the app’s sandbox entitlements file.

There are two ways a security-scoped bookmark can be created – either with read and write access, or read-access only (NSURLBookmarkCreationWithSecurityScope or it plus NSURLBookmarkCreationSecurityScopeAllowOnlyReadAccess).

I don’t know why this causes the issue, but changing the bookmark creation options from read and write access to read-access only fixed the issue (and moving the file is still possible with Yoink). It definitely looks like a sandbox bug to me.

Why should an app-internal bookmark (mind you, it’s only a reference to a file, not the file itself, created from an NSURL object) write to and change the webarchive’s file so that it a) can’t be opened anymore and b) shows the bookmark-creating-app as its creator?

Bug Reporting to Apple

I reported this bug to Apple (rdar://21765077) with an example project you can download here, if you’d like to see for yourself. Feel free to dupe the bug with Apple’s Bug Reporter Tool – I’d appreciate it :)

Now all that’s left to say is thank you for your patience, Christoph. You reported this issue at the beginning of March 2015 and had to wait until now for it to be fixed (and still a little longer because I have to submit the update to Apple for release on the Mac App Store as well). I really appreciate your continued feedback on the app!

Correction

Thanks to Michael Tsai for pointing out on his blog that .webarchive files aren’t bundles, as I falsely stated, but binary property lists.

What I… (June 2015)

Welcome to the 3rd instalment of “What I…” for June 2015 – again a little late this time. I blame having to dog-sit my girlfriend’s mom’s dog “Lucky” ;)

… Did

Dog Lucky

Dogsitting.
Lucky is with us every now and then, but having him here day and night for an entire week is a completely different experience.
For example, getting up at 5.45 sharp to take him out for a walk is something else compared to my typical routine.

ScreenFloat Update (click)
After successfully (and finally) releasing Glimpses, I’m able to move on to the next app that needs updating – ScreenFloat.
I have a couple of nice things planned. That will take some time, though, as I need to restructure some parts of the app, for example the storage of screenshots.
I don’t have an ETA yet and haven’t even started coding yet, I’m very much still in the planning stages and code-reviewing.

New App for Mac (and possibly iOS)
Another reason I haven’t started coding on the ScreenFloat update yet is that I’ve started working on a completely new app this month.
I like working on side-projects, it tends to restore my energies. Sometimes, working on an app that you’ve been working on for quite some time can become unexciting. At times, you want to do something else.
So projects I work on on the side are a good way to get my head out of what I should be working on and ‘reboot’ so I can return to, in this case, ScreenFloat with fully recharged batteries and a fresh eye at things.
Regarding the new app itself, it’s too soon to share anything about it, other than that progress is going quite well. I’ve been working on a custom UI for it and it has been taking a lot of time. It still is ;)

… Didn’t Do That I Had Planned

Post about ESSAudioWaveView
In my Mac app Glimpses, that turns your photos and music into stunning still motion videos, I use a piece of custom UI called ESSAudioWaveView, and it allows the user to select a portion of a song to be selected and used for the video.
I had planned to post about it this month but time wasn’t on my side.
It’s on my list, and perhaps I can do it in July. It will be a multi-part series.

Start coding work on ScreenFloat
Having gotten distracted with the new app I wrote about above, I didn’t get as much work done on ScreenFloat as I had hoped and wanted.
On the other hand, having updated Yoink to version 3, Transloader for iOS with a Today Widget and Action Extension, and Glimpses 2.0 in the last couple of months, I just needed some time to get away from my existing apps and do something else for a while.
A new app is always good therapy ;) 

… Downloaded

Gestimer Icon

Gestimer (click)
Developed by fellow Austrian Martin Nguyen, this Mac app sits in your menu bar and lets you quickly create short-time reminders (for example, take the dog for a walk in 30 minutes).
It has a great UI and I suggest you go check it out :)

LEGO Jurassic World Icon

LEGO Jurassic World (click)
As soon as it was announced, I knew I had to get it. And I wasn’t disappointed.
I played LEGO Star Wars, LEGO Indiana Jones and they were just plain fun and entertainment, with a unique style of humor.
The only thing that, “uhm, urr, staggers me” (inside joke here) is their use of in-movie-audio for dialogs. I often have a hard time understanding what is being said – perhaps they should have play-tested this a little longer and adjust the audio levels here.
Anyway, I’m at the beginning of the second movie now, ‘The Lost World’, and can’t wait to play again.

… Read

Link: The Process Behind Putting Firewatch On Stage At E3 (click)
Written by the Campo Santo team who are responsible for the game Firewatch which will be released some time in the future (in case you don’t know, the well-known Mac software company Panic, Inc. acts like a sort-of-publisher for the game), this blog post gives a look at what goes into a E3 demo. A very interesting read.

Link: An Oral History Of Deus Ex (click)
Fifteen years ago, Deus Ex was released. The girls and guys of Gamasutra sat down with the project director, lead programmer, composer and lead writer of the game, trying to “shed some light on how this seminal game was developed”.

Link: Postmortem: Blizzard’s Diablo II (click)
Also fifteen years ago, Diablo II was released – one of my (if not the) all-time favorite games I ever played. This is a post-mortem of Diablo II, written a couple of months after Diablo II was released, back in the year 2000.

We Don't Need Roads Book Cover

Book: We Don’t Need Roads – The Making of the Back To The Future Trilogy (click)
As the title says, a book about the making of the Back To The Future movie trilogy. Based on interviews with the people who made the movies, Robert Zemeckis, Bob Gale, Christopher Lloyd and countless others.
The cover looks great. I guess the way they figured, if you’re going to make a book cover, you better do it with some style!

… Listened To

Der Übercast Episode 32 (click)
A german podcast (alternatively with video). In this episode, the guest is Thorsten Lemke, the legendary developer responsible for Graphic Converter.

… Watched

WWDC Sessions (click)
Of course, as always with WWDC, there’s plenty of new stuff and with Apple posting the sessions as videos online, it’s very easy to get caught up with everything that’s coming.
My favorite sessions are:
– Session 102; Platforms Sate of the Union
– Session 506; Editing Movies in AVFoundation
– Sessions 603 and 607, 604, 606, 608; (What’s New in Metal, SpriteKit, SceneKit, GameplayKit) – Even though I don’t work on games right now, these sessions are always some of the first I watch
– Session 225; What’s New in NSCollectionView
– Sessions 704 and 715; What’s New in CloudKit and CloudKit Tips and Tricks

Images

Jurassic World (click)
What Jurassic Park sequels have taught me is to go into these movies with very, very, very low expectations. So I did. And what happened? I actually found it enjoyable.
Please do not read on if you haven’t seen the movie yet but plan to, there might be spoilers from here on.
Spoilers to come…
Still to come…
One more for good measure…

Here we go:
Forget the ‘love story’ – it’s totally inconsequential and unnecessary.
Forget the ‘brother story’ – it’s completely tacked-on.
Forget the characters – they’re totally underdeveloped.
Forget the ‘dinosaurs as weapons for the army story’ – sucks.
Forget the ’training Velociraptors like dogs’ thing – sucks big time.

With all that, I still found it to be enjoyable and entertaining. I can’t exactly put my fingers on it, but it’s got some good action, suspense and dinosaurs.
Perhaps it’s enjoyable to me because it’s the first ‘Jurassic’ movie I’ve seen in the cinema.

Dil Dhadakne Do Movie Cover

Dil Dhadakne Do (click)
A very funny comedy about a dysfunctional Punjabi family on a cruise trip.
My girlfriend wanted to see it and she was not disappointed – and neither was I. She loved the songs and dance numbers.

… Ate

Lemon Curd Berry TrifleLemon Curd Berry Trifle

… Went to See

IMG 9986Carnuntum, an archaeological site near Vienna, Austria

Guest Blog Post: The Story Behind Review Times

[ Note: This is a guest blog post written by Frank Gregor (@TheCocoaNaut on twitter), an OS X- and iOS developer based in Austria, about his Mac app Review Times ] Review Times Icon

Once upon a time… No, only fairytales start that way. And this is no fairytale. It is a “true story”, like, I’m sure, many other developers have experienced one way or another. (-:

Some time around Christmas 2014, I stumbled upon a tweet, linking to appreviewtimes.com. Many of you may know about this website already. It gives you the average time Apple is currently taking to review apps for submission on the iOS and Mac App Stores.

I thought, hey, that would make a nice little Mac tool. So I asked a couple of developers on Twitter what they thought of such an app. The response I got was great. Without exception, everyone I asked wanted something like that. I had had some free time on my hands, so the idea manifested.

I wanted to build a small app that would live exclusively in the menu bar and offered a Today Widget. The information I would display in the app I would get from appreviewtimes.com. So I contacted Dave Verwer (@daveverwer on twitter), founder of Curated and publisher of the well-known iOSDevWeekly newsletter. He was the contact listed for the Shiny Development team, who are the creators of appreviewtimes.

I asked Dave if they perhaps already had such an app available or in development, and if that wasn’t the case, if they’d mind if I developed the app and used their data for it:

Hey Dave,

I found you as contact for Shiny Development, the maker of App-Review-Times. So I think it would be a great thing to have a little OS X tool that shows in an app extension the current review times via Notification Center. I’m building exactly such a tool and I have to grab and parse the website to get all the needed data.

So, to make things easier for development: Would it be possible to get all that raw data (iOS & OS X) as JSON or XML? Would be really great! (-;

Cheers, phranck

Dave answered promptly:

Hi Frank

I’m afraid that we don’t have either JSON or XML feeds of the information.

Please feel free to scrape the HTML though, as long as you include a link back to the original site.

Thanks Dave

Great! Now nothing stood in the way of me selling this App on the App Store (though I would learn better later). I got to work and within two days, I had a working prototype.

The fact that this app would run as a menu bar tool was very beneficial to me. I had been working on a piece of open source code, which does exactly that: place an icon in the menu bar and display a popover when clicked. This gave me a very good reason to further develop and improve the open source code.

Screenshot of Review Times Popover

Now was the time to tweet some screenshots, to whet the appetite of potential customers. It worked well. I received a lot of inquiries about what the app was about. This way, I got my first beta testers.

Review Times Today Widget

All in all, about three to four weeks went into developing the app and I believed the app was ready. I needed a nice icon that should really catch the eye. I made the acquaintance of Dan (@derpixeldan on twitter) and he created something truly wonderful at an unbeatable price… (-;

Around Mid-January, I submitted the app to the Mac App Store. Excitedly I waited for the notification on my iPhone that the review process had begun. About three weeks later, I received the message I longed for.

It didn’t take long and the review team contacted me again. But this message wasn’t what I had hoped for. My app got rejected! The reason Apple gave me was

…featuring speculative information about Apple products/services…

which I didn’t find reasonable at all. I was shocked. Every developer who ever received a “Rejected” notice knows this feeling of unease. I was annoyed and angry at the same time. Why in the world… ?!?! Did I go about this entire thing too naively?

Appstore Rejection

If you are familiar with Apple, you know they like to have control. And speculative information – like the average time it may take for Apple to review an app until submission to the App Store, shown by an app by some developer – well, that was completely out of their control. So Apple showed me, who’s boss.

That took the wind out of my wings in respects to this tool. I didn’t want to get into a discussion with Apple, it wouldn’t have lead anywhere anyway. I vented on twitter and let the whole thing rest for a couple of days – until, again on twitter, I received inquiries about what had happened to the app and how I would proceed with it. There were two possible options for me: 1) publish it as open source code 2) sell it outside of the Mac App Store from my website

On Twitter, I asked for input.

Since previously, people had stated their interest in paying for this app, I decided to go with option 2. After some research and testing how to best implement selling it from my website, I came to the conclusion that I should use Paddle. Their integration into an app is absurdly easy to do and they have first-class support.

All said and done, it took three more days of coding and testing and then everything was ready. I built a one-page website, wrote to many Mac-Blogs and magazines (none of whom replied!) and announced the release on twitter. Since so many developers wanted this tool, I was preparing for incredible sales-days… (-:

Poppycock. Virtually nothing happened. I sold a couple of copies, but all in all it went very, very slowly. Even after two months – with heavy-duty tweeting about it – I didn’t even make up for the cost of the icon. Reality had its grip on me again. So I had to act. Without thinking about it too long, I deleted all license- and purchase-handling code and released the app for free from my website.

Now everyone can download and install the app from http://reviewtimes.cocoanaut.com. Since then, I’ve had about 1-2 downloads a day.


Frank Gregor (@TheCocoaNaut on twitter) is an OS X- and iOS developer based in Austria. Among others, he is responsible for the Mac apps Review Times, Nekrologger and the iOS app f4analyse.