Post-download script

View previous topic View next topic Go down

Post-download script

Post  ComicTagger on Sat Feb 09, 2013 7:57 pm

Here's a sample script for processing newly acquired comic files, similar to the one manders2600 developed, that will covert, tag, and rename the files. This one is in bash, so suitable for Mac or Linux only, probably. It uses the new export-to-zip feature, so a lot less file management. Suitable for running by hand, or via some other program.

Code:
#!/bin/bash
# post-processing script for comics.

#make bash globbing case-insensitive
shopt -s nocaseglob

# Where you want the processed comics to go
DEST="/data/downloads/tagged_comics"
LOGFILE="$DEST/log.txt"

# You might need to change this
COMICTAGGER="/Applications/ComicTagger.app/Contents/MacOS/ComicTagger"

#-----------------------------------------------------
log ()
{
   if [ -n "$1" ]
   then
      data="$@"
      echo "[$(date +"%D %T")] $data"  >> $LOGFILE
      echo "[$(date +"%D %T")] $data"
   else
      while read data
      do
         echo "[$(date +"%D %T")] $data" >> $LOGFILE
         echo "[$(date +"%D %T")] $data"
      done
   fi
}
#-----------------------------------------------------
touch $LOGFILE

# The first argument ($1) should be the folder with the comics
if [ "$1" == "" ]; then
   WORKDIR="."
else
   WORKDIR="$1"
fi

LOOK_FOR_COMICS=`ls "$WORKDIR"/*.cb[rz] 2> /dev/null`
if [ "$LOOK_FOR_COMICS" == "" ] ; then
   echo "No comics here in $WORKDIR" 2>&1 |log
   exit
fi

log  "#####################################################"
log " # Processing comics in $WORKDIR"
log  "#####################################################"

log  "-------------Converting CBRs to CBZ------------------"
$COMICTAGGER -e --delete-rar  "$WORKDIR"/*.cb[rz] 2>&1 | log
log  "-------------Tagging with CR style-------------------"
$COMICTAGGER -s -o -f  -m "year=" -t cr --nooverwrite --verbose "$WORKDIR/"*.cb[rz] 2>&1 | log
log "-------------Copying to CBI--------------------------"
$COMICTAGGER -c cr -t cbl --nooverwrite "$WORKDIR/"*.cb[rz] 2>&1 | log
log "-------------Renaming--------------------------------"
# Do the rename twice, since the filename overrides the internal metadata,
# and the tagging may have failed...
$COMICTAGGER -r -f  -t cr "$WORKDIR/"*.cb[rz] 2>&1| log
$COMICTAGGER -r  -t cr "$WORKDIR/"*.cb[rz] 2>&1| log

# Don't move anything if we're working in the current directory
if [ "$WORKDIR" == "$1" -a "$WORKDIR" != "." ]; then
   log "-------------Relocating------------------------------"
   mv  "$WORKDIR/"*.cb[rz] $DEST 2>&1 | log
   rmdir "$WORKDIR" 2>&1 | log 
fi
log "Done!"
avatar
ComicTagger
Admin

Posts : 208
Join date : 2012-12-02

View user profile http://comictagger.forumotion.com

Back to top Go down

Making it a launchd service

Post  shadowcharly on Wed Feb 12, 2014 11:24 am

Hi, i've been tinkering with comictagger for an automated tagging of my comics, and with the excellent help of your script i've managed to create a system that works fine:

This works with OSX 10.9 (and i suppose with earlier versions too) or anything that uses the launchd daemon.

We are gonna use a modified version of the script to tag our comics, and then we are gonna set up a daemon that will run periodically or whenever a new folder is detected in the _unTagged folder:

The Script:
Code:

#!/bin/bash
# post-processing script for comics.

#logging function Very usefull
#-----------------------------------------------------
log ()
{
   if [ -n "$1" ]
   then
      data="$@"
      echo "[$(date +"%D %T")] $data"  >> $LOGFILE
      echo "[$(date +"%D %T")] $data"
   else
      while read data
      do
         echo "[$(date +"%D %T")] $data" >> $LOGFILE
         echo "[$(date +"%D %T")] $data"
      done
   fi
}

#-----------------------------------------------------


# Where you want the processed comics to go
DEST="/Where/do/we/put/our/tagged/files"
LOGFILE="$DEST/_ComicTagger_log_new.txt"

# Check to see if we are already working (we just want one process working at the same time
LOCKFILE="$DEST/comicTaggerLock"

# where's the find command
FIND="/usr/bin/find"
GREP="/usr/bin/grep"
CUT="/usr/bin/cut"

log "--------------------STARTING------------------"

if [ -f "$LOCKFILE" ]; then
   log "Shit, we're second!"
   exit
else
   log "Lets stablish a beach head"
   touch "$LOCKFILE"
fi

#make bash globbing case-insensitive
shopt -s nocaseglob


# You might need to change this
COMICTAGGER="/Applications/ComicTagger.app/Contents/MacOS/ComicTagger"

touch $LOGFILE


# The first argument ($1) should be the folder with the comics
if [ "$1" == "" ]; then
   WORKDIR="."
else
   WORKDIR="$1"
fi


# step 1, search for untagged files in the workdir
#Set the field separator to new line
IFS=$'\n'

UNTAGGED=`$FIND "$WORKDIR" -name "*.cb*" -exec $COMICTAGGER -p --terse {} + |$GREP "\[ none \]" |$CUT -f1 -d:`
log "-------Files to Process----"
if [ "$UNTAGGED" ]; then
   log "Hay ficheros que taggear"
#    log "$UNTAGGED"
else
   log "No hay ficheros que taggear"
fi

#Step 2, tag files as UNTAGGED
log "-------Let's tag this files------"
# log "------------ Renaming Untagged comics for further processing --------------"
# log "$UNTAGGED"
for i in $UNTAGGED
do
    if [[ $i == *\(UNTAGGED\)* ]]
    then
        log "$i: No need to rename";
    else
        NEWNAME="${i%.*} (UNTAGGED).${i##*.}"
        mv $i $NEWNAME
#         log "\"$i\": renamed to " \"$NEWNAME\"
        #echo "$i: Need to Rename";
    fi
done

#step 3, process just the files with the untagged tag
log  "#####################################################"
log " # Processing untagged comics in $WORKDIR"
log  "#####################################################"
# Convert to cbz if needed
$FIND "$WORKDIR" -name "*.cbr" -exec $COMICTAGGER -e --delete-rar {} + 2>&1 #| log

# Tag just the untagged ones
$FIND "$WORKDIR" -name "*\(UNTAGGED\)*" -exec $COMICTAGGER -s -o -f  -m "year=" -t cbl --nooverwrite --noabort --verbose  {} + 2>&1 #| log

# Step 4, rename files to proper series and number, etc...
# Renaming
log "-------------Renaming--------------------------------"
$FIND "$WORKDIR" -name "*.cb*" -exec $COMICTAGGER -r -f  -t cbl   {} + 2>&1 # | log

#Double Renaming
$FIND "$WORKDIR" -name "*.cb*" -exec $COMICTAGGER -r  -t cbl  {} + 2>&1 # | log

# Step 5, tag again the untagged ones
#Set the field separator to new line
IFS=$'\n'
UNTAGGED=`$FIND $WORKDIR -name "*.cb*" -exec $COMICTAGGER -p --terse {} + |"$GREP" "\[ none \]" |"$CUT" -f1 -d:`
# log "let's refind the untagged ones------"
# log "$UNTAGGED"
for i in $UNTAGGED
do
    if [[ $i == *\(UNTAGGED\)* ]]
    then
        log "$i: No need to rename";
    else
        #we add an (UNTAGGED) to the filename so we can easily check for untagged comics
        NEWNAME="${i%.*} (UNTAGGED).${i##*.}"
        mv $i $NEWNAME
        #log "\"$i\": renamed to " \"$NEWNAME\"
        #echo "$i: Need to Rename";
    fi
done

# Step 6, if all comics tagged, move them to _tagged
EMPTYFOLDER=`$FIND $WORKDIR -name "*.cb*"`

# Very important to use the variables for the binaries, so when called from launchd, they get the right result.
UNTAGGED=`$FIND $WORKDIR -name "*.cb*" -exec $COMICTAGGER -p --terse {} + |"$GREP" "\[ none \]" |"$CUT" -f1 -d:`


# log "--------Final Untagged files---------"
# log "$UNTAGGED"
# log "--------End of untagged files--------"

# Check for not null string as a result of finding cb* files in workdir

# TO-DO : Detect which folders are completely tagged and which are not, so we can move them individually
if [ "$EMPTYFOLDER" ]; then
   if [ ! "$UNTAGGED" ] ; then
      log "-------------Relocating------------------------------"
      mv "$WORKDIR/"* $DEST 2>&1 | log
   else
      #not completely tagged, leave the files for further process
      log "--------- Not Relocating--------"
   fi
fi

rm "$LOCKFILE"
log "-------------------FINISHED--------------------"


I made some changes to the original script so it works with multiple folders and subfolders, and i use ComicBookLover, so i modified that part too. I may need to change the "no comics in folder" part too, but right now it works ^.^
EDIT: Some more edits to be able to invoke it as a daemon

I saved the script as "tagcomics" in my /opt/local/bin and chmod'd it to +x
> sudo chmod +x tagcomics
> sudo mv bagandtag /opt/local/bin/tagcomics


The Daemon setup
The setup for the daemon is pretty simple, really, but can be confusing. We need to create a .plist file that defines our daemon:
Code:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>ExitTimeOut</key>
 <integer>600</integer>
 <key>RunAtLoad</key>
 <true/>
 <key>Label</key>
 <string>com.Whatever.comicTaggerScript</string>
 <key>ProgramArguments</key>
 <array>
 <string>/opt/local/bin/tagcomics</string>
 <string>/Path/to/untagged/folder</string>
 </array>
 <key>StartInterval</key>
 <integer>3200</integer>
 <key>StandardOutPath</key>
    <string>/var/log/comicTagger.log</string>
    <key>StandardErrorPath</key>
    <string>/var/log/comicTagger.log</string>
    <key>Debug</key>
    <false/>
</dict>
</plist>

And we save it to /Users/myUserName/Library/LaunchAgents/com.wathEverSuitsYou.comicTaggerScript.plist
and we have to assign it the proper privileges.(unable to post links yet, just google it Wink)
> sudo chown root /Users/myUserName/Library/LaunchAgents/com.wathEverSuitsYou.comicTaggerScript.plist
> sudo chmod 644 /Users/myUserName/Library/LaunchAgents/com.wathEverSuitsYou.comicTaggerScript.plist

Make sure you get the correct plist parameters before changing the permissions.

If you need to tinker with the plist, more info at:(link)
type man launchd at terminal


Once everything is set to your liking, you just have to get it started up with:
>sudo launchctl load /Users/myUserName/Library/LaunchAgents/com.wathEverSuitsYou.comicTaggerScript.plist
And when you need to edit the .plist file, you first have to
> sudo launchctl unload /Users/myUserName/Library/LaunchAgents/com.wathEverSuitsYou.comicTaggerScript.plist

And that's all, i think.

EDIT: Thank you very much to comictagger for his help in the modifications to the script, that line that finds the untagged comics, its his magic, boys! ^.^
EDIT2: Added some more refination to the script, it works better when periodically checking for tagged files, we just search for the ones that are still untagged. Still not improved in the all or nothing logic used to move the folders with the cbz files.

shadowcharly

Posts : 3
Join date : 2014-02-12

View user profile

Back to top Go down

Would love a windows version

Post  dlrdlrdlr on Wed Apr 01, 2015 5:16 pm

All these options seem pretty linux and OSX specific and I was hoping there would be a windows option, I don't know enough about scripting to do it myself but I know this code worked for me when ran manually from the cmd

Code:
./comictagger.exe -s -t cr -f -o $DOWNLOADED_COMIC

dlrdlrdlr

Posts : 2
Join date : 2015-04-01

View user profile

Back to top Go down

A Windows PowerShell Version

Post  tomdelise on Thu May 21, 2015 1:32 am

Below is some code that works in Windows PowerShell.  I'm not a PowerShell programmer so I don't know what version this will or will not work with .  I wrote it in 8.1.
It will work very much like the script at the top of this page but I like to put my folders into a folder and process them all so the main essentially traverses the sub-folders in the folder that you define either as a parameter or your current folder.

If you want to use it like the first post from comictagger, I think it will work if you just change the main call at the bottom of the script to processComics and pass the source variable (haven't actually tried it)

Code:

# Param([string]$source=".")
# Unfortunately the line above must be the first line if you want to use a named param like -source
# If you do not define your source in your call, it will default to your current directory

# Global variables
# I'm lazy and redefine source here for testing, you can uncomment this and hardcode it here if you want
#[string]$source="D:\comics\unprocessed"
# Set your destination of where the processed comics will go
[string]$dest="D:\comics\processed"
# Set to where you want your log file written - currently set to be created where your comics are sent
[string]$logfile="$dest\log.txt"
# This starts the logging of all messages
Start-Transcript -path $logfile


# Function to process the current working directory that is passed to it.  
# See comments below on how the main processes the folder
function processComics($WORKDIR) {
  
   # Set this to the path of your executable - in Windows it is usually here
   [string]$comictagger="C:\Program Files (x86)\ComicTagger\comictagger.exe"
  
   # Use Write-OUtput if you want to input text to the std out and to the log
   # Notice how two lines down you can insert your variable directly into the quotes
   Write-Output  "#####################################################"
   Write-Output " # Processing comics in $WORKDIR"
   Write-Output  "#####################################################"
   Write-Output  "-------------Remove Text Files------------------"
  
   # This will remove the .txt files that are sometimes in the folders that catalog what is in the folder
   Remove-Item "$WORKDIR/*.txt" -recurse -force | Out-Host
  
   # Remove the Archive attribute if it is set on the files in this folder
   # it will change all read onlys in the folder if you process it
   removeReadOnly $WORKDIR
  
   Write-Output  "-------------Converting CBRs to CBZ------------------"
   # This is pretty much the same as the other scripts but I do not convert to comic lover, change as needed but follow format
   & $COMICTAGGER -e --delete-rar "$WORKDIR/*.cb*" 2>&1 | Out-Host
   Write-Output  "-------------Tagging with CR style-------------------"
   & $COMICTAGGER -s -o -f -t cr -1 --nooverwrite --verbose --noabort "$WORKDIR/*.cb*" 2>&1 | Out-Host
   Write-Output "-------------Renaming--------------------------------"
   #  Do the rename twice, since the filename overrides the internal metadata,
   #  and the tagging may have failed...
   & $COMICTAGGER -r -f -t cr "$WORKDIR/*.cb*" 2>&1| Out-Host
   & $COMICTAGGER -r -t cr "$WORKDIR/*.cb*" 2>&1| Out-Host
   Write-Output "-------------Relocating------------------------------"
   # Moves all of the files to your destination
   Move-Item -Path  "$WORKDIR/*.cb*" -Destination $DEST 2>&1 | Out-Host

   # Call the funciton to remove the directory if it is empty
   removeEmptyDirectory $WORKDIR
  
}

# Function will check the folder to see if it is empty and if so will remove it
function removeEmptyDirectory($WORKDIR){
   # Checks if the folder is empty of files if it is it will remove the folder
   If((Get-ChildItem $WORKDIR -force | Select-Object -First 1 | Measure-Object).Count -eq 0)
   {
        Remove-Item $WORKDIR | Out-Host
   }
   Else
   {
        Write-Output "***************************************************************************************************************"
        Write-Output "*********************** $WORKDIR IS NOT EMPTY ***********************************"
        Write-Output "***************************************************************************************************************"
   }

}


# Will set all the attributes in the current working folder to normal - removing any read only attributes
# This will not recurse so it will only do the current working directory and no subs of that directory - hopefully safer
function removeReadOnly($WORKDIR){
    
    Get-Item $WORKDIR | foreach {$_.Attributes = 'Normal'}

}


function main(){

    # Grab a collection of all folders within your source and loop through them checking for comics in each one
    $dirs = Get-ChildItem $source | Where-Object { $_.Attributes -band [System.IO.FileAttributes]::Directory }
    foreach ($dir in $dirs)
    {
       $fullpath=$dir.FullName + "\*.*"
       # Get count of how many comic files ar in the directory
       $comic_count=(Get-ChildItem $fullpath -include *.cbr,*.cbz).Count
       Write-Output "the count of comics in $dir.FullName is $comic_count"
       # If there are any comics in the folder, pass the foldername to the processComics function
        # If not try to remove the directory if it is empty
       If ($comic_count -gt 0) {
          processComics $dir.FullName
            #removeReadOnly $dir.FullName
       }
       Else
       {
            removeEmptyDirectory $dir.FullName
       }
    }
    # Stop and close the log file
    Stop-Transcript
}

main

tomdelise

Posts : 2
Join date : 2015-05-18

View user profile

Back to top Go down

Re: Post-download script

Post  tomdelise on Thu May 21, 2015 11:51 pm

I cleaned up the script a little so it ins't as bad formed as it was. I also was going to play with the great move2folder script in the next thread that takes the tagged comics and creates directories and sub-directories from that BUT I didn't have to. The script will work as it is written in PowerShell if you have python 2.7 installed. I removed the check that will write out an error if in windows since it works just fine.

The cleaned up script uses 3 folders. One to place the unprocessed comics that are in their own folders. It will tag the comics and move them all into a processed folder then the move2folder will create all the folders and subs in the final destination folder moving the files if they are tagged and there isn't a copy already there.

You can also setup this to work like the post above by using Windows Task Scheduler. If you put all of your unprocessed comic folders into your unprocessed folder you can run this nightly from your scheduler. The only thing that I believe is still a problem is getting the script to pause when your ComicVine ID locks for a bit.

Code:

# Param([string]$source=".")
# Unfortunately the first line has to be there if you want to use a named param like -source
# If you do not define your source in your call, it will default to your current directory

# Global variables
# I'm lazy and redefine source here for testing, you can uncomment this and hardcode it here if you want
[string]$source="D:\comics\unprocessed"
# Set your folder of where the tagged comics will go
[string]$middle="D:\comics\processed"
# Set the folder of where the tagged comics will go and create folders from the tags
[string]$finalDest="D:\comics\comicsdone"
# Path to where the move2folder or other post run script you want to run is
[string]$scripts="D:\comics\privatescripts"
# Set this to the path of your executable - in Windows it is usually here
[string]$comictagger="C:\Program Files (x86)\ComicTagger\comictagger.exe"
# Set to where you want your log file written - currently set to be created where your comics are sent
[string]$logfile="$dest\log.txt"




# Function to process the current working directory that is passed to it. 
# See comments below on how the main processes the folder
function processComics($workdir) {
 
 
  # Use Write-OUtput if you want to input text to the std out and to the log
  # Notice how two lines down you can insert your variable directly into the quotes
  Write-Output  "#####################################################"
  Write-Output " # Processing comics in $workdir"
  Write-Output  "#####################################################"
  Write-Output  "-------------Remove Text Files------------------"
 
  # This will remove the .txt files that are sometimes in the folders that catalog what is in the folder
  Remove-Item "$workdir/*.txt" -recurse -force | Out-Host
 
  # Remove the Archive attribute if it is set on the files in this folder
  # it will change all read onlys in the folder if you process it
  removeReadOnly $workdir
 
  Write-Output  "-------------Converting CBRs to CBZ------------------"
  # This is pretty much the same as the other scripts but I do not convert to comic lover, change as needed but follow format
  & $comictagger -e --delete-rar "$workdir/*.cb*" 2>&1 | Out-Host
  Write-Output  "-------------Tagging with CR style-------------------"
  & $comictagger -s -o -f -t cr -1 --nooverwrite --verbose --noabort "$workdir/*.cb*" 2>&1 | Out-Host
  Write-Output "-------------Renaming--------------------------------"
  #  Do the rename twice, since the filename overrides the internal metadata,
  #  and the tagging may have failed...
  & $comictagger -r -f -t cr "$workdir/*.cb*" 2>&1| Out-Host
  & $comictagger -r -t cr "$workdir/*.cb*" 2>&1| Out-Host
  Write-Output "-------------Relocating------------------------------"
  # Moves all of the files to your destination
  Move-Item -Path  "$workdir/*.cb*" -Destination $middle 2>&1 | Out-Host

  # Call the funciton to remove the directory if it is empty
  removeEmptyDirectory $workdir
 
}

# Function will check the folder to see if it is empty and if so will remove it
function removeEmptyDirectory($workdir){
  # Checks if the folder is empty of files if it is it will remove the folder
  If((Get-ChildItem $workdir -force | Select-Object -First 1 | Measure-Object).Count -eq 0)
  {
        Remove-Item $workdir | Out-Host
  }
  Else
  {
        Write-Output "***************************************************************************************************************"
        Write-Output "*********************** $workdir IS NOT EMPTY ***********************************"
        Write-Output "***************************************************************************************************************"
  }

}


# Will set all the attributes in the current working folder to normal - removing any read only attributes
# This will not recurse so it will only do the current working directory and no subs of that directory - hopefully safer
function removeReadOnly($workdir){
   
    Get-Item $workdir | foreach {$_.Attributes = 'Normal'}

}


function main(){

    # Grab a collection of all folders within your source and loop through them checking for comics in each one
    $dirs = Get-ChildItem $source | Where-Object { $_.Attributes -band [System.IO.FileAttributes]::Directory }
    foreach ($dir in $dirs)
    {
       $fullpath=$dir.FullName + "\*.*"
       # Get count of how many comic files ar in the directory
       $comic_count=(Get-ChildItem $fullpath -include *.cbr,*.cbz).Count
       Write-Output "the count of comics in $dir.FullName is $comic_count"
       # If there are any comics in the folder, pass the foldername to the processComics function
        # If not try to remove the directory if it is empty
       If ($comic_count -gt 0) {
          processComics $dir.FullName
            #removeReadOnly $dir.FullName
       }
       Else
       {
            removeEmptyDirectory $dir.FullName
       }
    }
   
   
    # Run comictagger with the python script to create folders and subfolders based on tags
    & $comictagger -S "$scripts\move2folder.py" "$middle" "$finalDest" 2>&1| Out-Host
   
}

# This starts the logging of all messages
Start-Transcript -path $logfile

main

# Stop and close the log file
Stop-Transcript

tomdelise

Posts : 2
Join date : 2015-05-18

View user profile

Back to top Go down

Re: Post-download script

Post  Sponsored content


Sponsored content


Back to top Go down

View previous topic View next topic Back to top


 
Permissions in this forum:
You cannot reply to topics in this forum