The problem, in a nutshell, is that image which you might take as experiments or, in my case, photo tests, end up as images shown off on the main Photos live tile (e.g. below, left), which cycles through a 'random' selection of photos. It's not exactly a show-stopping problem, but it is an annoyance.
What's happening is that at some point Windows Phone takes a look at your settings and picks out some random images to use - and caches them - and then continues to cycle through these even if the source images are subsequently deleted. What's needed is a way to force Windows Phone to choose some different images, ones that you actually like and kept.
Go into Settings and swipe to the side to 'applications', then pick 'photos+camera':
Under 'Start Tile shows', pick a different option, then step 'back' and try the Start screen again. In my case, I prefer to filter my large photo collection anyway, so setting the pick list to 'favourite photos' achieves two objectives - it forces a refresh of the Photos live tile cache and also sets my Start screen up to just show my all time favourite images. Well, those physically on this particular phone, anyway.
Of course, if you wanted a wider selection, you can then go back into the same dialog and pick 'all photos' again - the cache is rebuilt for a second time and you should get a different selection picked out, with all your old (/deleted) images no longer present. Phew!
Now.... anyone know how to configure what's shown on the Lumia Storyteller live tile?