Add new export/process/reimport command (was Resized Library)

[engeugenius]

Hello,
I would like to propose you a new feature.

Background: lot of people are using iCloud Photo Library - often shared with family -, and iCloud rates have also grown. Moreover, 95% of picture are food or useless stuffs.

Feature proposal: It would be nice to be able to export all library with high res photo, as we are already able to do, into our filesystem and replace each - or filtered group of - photo in the Library (i.e. iCloud) with a resized version (also containing the EXIF data, with geotagging), occuping less space.
Resize parameters and compression level can be chosen via command line.

Currently I've done this manually with Photos export feature, but it not convenient and error prone.

Thanks a lot,
Eugenio

[RhetTbull]

I was chatting with @oPromessa about something similar in #907 -- he's exporting old MPG files, converting to MOV and re-importing while preserving metadata. I think a general purpose "export - process - reimport with metadata" command would be helpful. In your case, the command could resize photos, in his case, convert files, etc.

Not sure what a good name would be (open to ideas!) but maybe something like osxphotos expim (for EXport Process IMport):

osxphotos expim --selected --command "convert {filepath} -resize 100x100 {filepath.stem}_resized{filepath.suffix}"

which would do the following:

1. Export the original image to a temporary directory
2. Run your command (and more than one command could be specified to chain them together)
3. Re-import the new file and apply all old metadata
4. Stage the original for deletion in a "To be deleted" folder (osxphotos cannot actually delete photos from Photos)

A few thoughts:

• Persons cannot currently be handled this way as there's no programmatic way to set them.
• How to handle photos with edited images? Ideally you wouldn't want to lose the original and associated edit but this complicates the process.

[engeugenius]

That's great, you got my ideas!
For persons, i think they can be recognized once again (manually) after expim command.. not a big deal, because you have to do it only once.
For edited images, i would suggest only to export the original one and the edited image as a "new" photo. Actually I would not care about this.

For command name, I would suggest something like "shrinklibrary", or anyway something more verbose than expim.
My two cents!

Unfortunately, i'm not into python world (only embedded C and C++, and some C# as hobby)

[RhetTbull]

For command name, I would suggest something like "shrinklibrary", or anyway something more verbose than expim.

Unfortunately that would only describe one possible use cases. I'm thinking of a more general tool that could be used any time you want to replace a photo in Photos while preserving the metadata. For example, another use case would be to strip the live video from live photos (see the link for an example script I'd previously done for osxphotos.

Other ideas for the command name:

• reimport
• reprocess
• process
• modify
• transform

For shrinking photos another option would be to covert jpeg and other formats to HEIC. osxphotos already has code for converting HEIC to JPEG using MacOS native calls (with GPU acceleration) and this could easily be used to go the other way too.

[rizwank]

Stripping live video, converting RAW+JPEG to JPEG, converting RAW to JPEG, convert to HEIC... so many things I'd love to do with this.

[RhetTbull]

@rizwank yes! I'm thinking the interface would have some "pre-canned" actions as well as the option to run any arbitrary command or call another python function the user supplies (e.g. a plug-in). For example:

--strip-live
--strip-original (strips original if photo is edited)
--strip-raw
--convert jpeg (convert to jpeg0
--convert heic (convert to heic, this would shrink jpeg files)
--convert mov (convert video to QuickTime)
--command "convert {filepath} -resize 50% {filepath}" (uses ImageMagick to rescale, but could be any command)
--function my_user_function.py::function (calls your function with the image file and re-imports the output of your function)

[oPromessa]

My vote on osxphotos transform

PS: or maybe osxphotos convert

[RhetTbull]

Once nice feature might be ability to export (using osxphotos export) then re-import a low-res version but include in the metadata for the low-res version a link to the exported original. This would allow you to "archive" photos while maintaining the album structure and other metadata to trim your iCloud usage. This could be comined with an AppleScript that could be attached to a shortcut such that activating the shortcut while a photo is selected in Photos, finds the original exported version and reveals it in Finder or opens it in Preview.

The original export path could be stored in the Photo's metadata (e.g the description) or in a separate key-value store to map re-imported photo to exported original.

[engeugenius]

...or export to a new HiRes (offline) Library containing the original photos.

[RhetTbull]

...or export to a new HiRes (offline) Library containing the original photos.

Do you mean to a new Photos Library? That might be better (certainly easier to implement) as a two-step process:

1. Export to a new library (I've already been thinking about a split or migrate (edit: maybe copy is better since osxphotos cannot actually delete or move any files from the source library) command that would move some photos to another library, either a new one or an existing library)
2. Reimport the low res versions into the original library.

Doing this requires using AppleScript and only one library can be scripted at a time (and switching libraries requires user interaction) so it will have to be multi-step no matter how implemented.

[rizwank]

Curious if this is still something you are considering. I'd personally love it; to be able to downres / remove combo RAW/JPG etc.

[RhetTbull]

@rizwank yes it's still on my list. The key to this is to be able to call the Apple APIs to request a change to the library. I have that prototyped but I'm working on figuring out how to use some undocumented private APIs so this can work with any library, not just the system library. I have that partially working.

[rizwank]

You are amazing. I don’t know if getting more users is important to you, but I see this particular workflow ending up on the various Apple photos blogs with a lot of interest.

…

On Tue, Aug 29 2023 at 12:57 AM, Rhet Turnbull ***@***.***> wrote: @rizwank <https://github.com/rizwank> yes it's still on my list. The key to this is to be able to call the Apple APIs to request a change to the library. I have that prototyped but I'm working on figuring out how to use some undocumented private APIs so this can work with any library, not just the system library. I have that partially working. — Reply to this email directly, view it on GitHub <#909 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABWPZA2QA6VI7YF5V25JUTXXUO5NANCNFSM6AAAAAATZRWJZY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[RhetTbull]

@rizwank thanks. I'm currently working on implementing a push-exif command (#160) to be able to push metadata from Photos back to the original photo in the library (this won't work on iCloud photos but will work for those that don't use iCloud and want to keep metadata synced to their originals. When done with that I'll take a look at this issue.

As an aside, I get about 10 hours/week to work on osxphotos (it's a hobby) and generally work on things in the following order:

1. Bugs that could cause data loss/corruption
2. Features I need for my own workflow
3. Things users really want (this is really hard to know as most issues have only 1 or 2 people commenting)
4. Things I find interesting. Many osxphotos features that are useful like the templating system came about only because I thought "that sounds interesting to work on".

[RhetTbull]

Done with osxphotos push-exif (in release v0.63.0) so now I'm going to think about this some more. For photos that have been edited, I need to figure out the workflow...do I make both available to the transformation function/command (e.g. {filepath} and {filepath_edited} or let the user choose which to keep? If both, I'll need to get the PhotoKit code working to request changes. This is relatively easy on the System Library but requires calls to undocumented private APIs for non-System libraries. What about RAW+JPEG? What about live Photos?

Some built in workflows wouldn't be hard but thinking about how to expose this transform capability to the user is more difficult for anything that's not an original photo or video with a single asset.

Happy for any feedback on how people might want to actually use this capability. I think this will be very powerful once implemented but getting the details right is tricky.

[RhetTbull]

I've made progress on hacking the Photos private APIs. I can now fetch, export, and delete assets from any library. Deleting will pop up a dialog for confirmation but I can bulk delete with only a single dialog. Now on to editing!

[RhetTbull]

I can now add assets to the library without resorting to AppleScript but so far only for the system libary. Still working on reverse engineering the private APIs for adding.

>>> from osxphotos.photokit import PhotoLibrary
>>> pl = PhotoLibrary()
>>> pl.add_photo("/private/tmp/test.jpeg")
'6526CC8B-6EA4-4CE8-8DDC-22168008F5DE/L0/001'
>>>

And it should be straightforward to add live photos, raw+jpeg, videos this way so I can do a round-trip from Photos -> transform -> Photos for any asset type.

[RhetTbull]

Have import working for all libraries not just system libraries and can also now create libraries.

[RhetTbull]

I'm making good progress on the new code that will let me have near native access to the entire Photos library and this be able to implement this feature. I'm take a short pause to work on adding iPhoto support to osxphotos (see #40) because I have a personal need for this (reading an old iPhoto library on Ventura, which no longer supports iPhotos). Once done I'll get back to this. What are the "transform" features you would find most useful? I think --strip-live would be great as I often shoot in live unintentionally. Other ideas?

[ulilicht]

Hi I saw your questions from some time ago:

What are the "transform" features you would find most useful? I think --strip-live would be great as I often shoot in live unintentionally. Other ideas?

Here are my 2cents:

Live Photos

• Live photos are mainly relevant if there is a person on the picture. How I use your "strip-live" Gist currently is that I have a smart album which looks for all live photos without faces, which are older than 3 months and not in an album. These are the ones which for me are safe to strip the live part from. When I strip live, I select all images in the smart album and run the transformation. I assume the current available filter logic would allow me to do this in osxphotos as well, without the smart album.

Videos

• I have quite some videos from pre-smartphone age with bad quality and bad compression. I'd love to bring them to a modern format --compress-video would be a great transformation.

Strip unneeded stuff

• as mentioned above already, --strip-raw and --strip-original would be great

Compression for unsorted images
My workflow is usually:

• Take pictures with smartphone or DSLR. Import them somehow, e.g. iCloud or through USB connected SD Card.
• Sort them into Albums after some time.

I am thinking about image compression for the pictures which are not in an Album after some time, e.g. resize all images to 2048 pixels given some rules, e.g.: Not in an Album and older than 3 Months.

This would leave me with high quality for the important ones, and acceptable quality for the rest.

[RhetTbull]

Thanks @ulilicht these are good use cases. This feature is on my roadmap but it's likely to be early next year before I can get this implemented).

[danVnest]

+1 for this!

In the meantime I put together this script using osxphotos and ffmpeg specifically for finding the largest videos in my library, compressing them, and reimporting.

Hopefully useful to others too. Do be careful though - not documented properly and limited testing (I've done about 100 videos so far).

[RhetTbull]

Thanks @danVnest that looks handy! I've been working on another companion library to osxphotos that would make this type work easier as it uses the native PhotoKit interface for interacting with the library and editing / replacing assets. It uses some private / undocumented features though (the public API is very limited) and this means it takes a fair bit more time for reverse engineering and testing. Once I get this done I should be able to implement this feature.

[danVnest]

That's great to hear @RhetTbull - good luck with the reverse engineering!