Comictagger and Mylar

View previous topic View next topic Go down

Comictagger and Mylar

Post  manders2600 on Tue Dec 11, 2012 4:29 pm

Saw this project and got extremely excited!

Are you aware of a project called "mylar" (a kind of sickbeard for comics)?

I would love to get your thoughts on possibly running comictagger and populating some of the command line options with comicvine data scraped by mylar.

Something like:

1. Mylar finds wanted comic and sends the nzb to sabnzbd,

2. Check to make sure CBR file is actually RAR archive
(checktype.pl, using malor89's perl scripts )

3. Convert .cbr to .cbz files:

mv comic.cbr comic.rar
unrar x comic.rar
zip -r comic.zip /comic
mv comic.zip comic.cbz

4. Tag with comictagger.py (command line metadata list generated from comicvine info already collected by mylar)

manders2600

Posts : 12
Join date : 2012-12-11

View user profile

Back to top Go down

Re: Comictagger and Mylar

Post  ComicTagger on Tue Dec 11, 2012 5:39 pm

I have a played a bit with Mylar, and it looks pretty cool, but I didn't have much luck getting it working. Not sure if it had to do with the index sites, or whatever. Looks like a lot of promise there, though, and certainly some synergy is possible with ComicTagger, especially given that they are both written in Python.

I guess if you set up sabnzbd to run a script after downloading a comic, you could do most of what you mentioned right now. The only trick would be getting the metadata from Mylar to ComicTagger. You would have to get Mylar to spit out the metadata into a file, and have it ready to match with the file that sabnzbd downloaded. Is it possible to do something like this already? Then you have to massage the data in the file to be a format that you could pass on the command-line to ComicTagger.



Another way, which would be a little redundant with respect to metadata downloads, but would be a *lot* easier, would be to let ComicTagger just re-fetch the metadata itself.

Assuming the the filename is parseable for series and issuenumber, just do this:
Code:

./comictagger.py -s -t cr -f -o  $DOWNLOADED_COMIC

I have been meaning to experiment with Mylar a bit, but in the wake of recent indexer shutdowns, I was wondering if it was even working anymore?
avatar
ComicTagger
Admin

Posts : 208
Join date : 2012-12-02

View user profile http://comictagger.forumotion.com

Back to top Go down

Re: Comictagger and Mylar

Post  manders2600 on Tue Dec 11, 2012 7:08 pm

ComicTagger wrote:I have a played a bit with Mylar, and it looks pretty cool, but I didn't have much luck getting it working. Not sure if it had to do with the index sites, or whatever. Looks like a lot of promise there, though, and certainly some synergy is possible with ComicTagger, especially given that they are both written in Python.

I guess if you set up sabnzbd to run a script after downloading a comic, you could do most of what you mentioned right now. The only trick would be getting the metadata from Mylar to ComicTagger. You would have to get Mylar to spit out the metadata into a file, and have it ready to match with the file that sabnzbd downloaded. Is it possible to do something like this already? Then you have to massage the data in the file to be a format that you could pass on the command-line to ComicTagger.


I assume it would be possible to have mylar spit out the data into a temp file or something, but I have basically zero experience with python, so I haven't really checked to see how this actually works. Certainly, the variables needed would be stored in mylar's database, but I don't know where they would be logically stored once the nzb is snatched, but before it is processed (in other words, how to tag the file prior to mylar's post-processing, or how to determine which specific file to tag after mylar's post-processing dumps it into a folder with other files).

ComicTagger wrote:Another way, which would be a little redundant with respect to metadata downloads, but would be a *lot* easier, would be to let ComicTagger just re-fetch the metadata itself.

Assuming the the filename is parseable for series and issuenumber, just do this:
Code:

./comictagger.py -s -t cr -f -o  $DOWNLOADED_COMIC
Yeah, that should work in most cases (though I think one would still need to convert .cbr to .cbz to be effective, since I believe comicbooklover stores its comicbookinfo in the zip file comment). It also requires the file name to be parseable, as you said, and sometimes that gets a little hairy.

If comictagger could use the information mylar had when it grabbed the file as indexed, it might work a little better (and the "scene exceptions" or naming issues for different comics could be centralized).

ComicTagger wrote:I have been meaning to experiment with Mylar a bit, but in the wake of recent indexer shutdowns, I was wondering if it was even working anymore?

Mylar was set up for newznab providers first, with other providers being an experimental feature. So it still works, even with nzbmatrix and newzbin shutting down. Also, since anyone can basically run a newznab indexer on their local machine if they want to, it would be really hard for this to get shut down.




manders2600

Posts : 12
Join date : 2012-12-11

View user profile

Back to top Go down

Re: Comictagger and Mylar

Post  ComicTagger on Tue Dec 11, 2012 8:48 pm

The filename parsing for ComicTagger is actaully pretty robust with respect to "scene"-style names, as that's what a lot of my testing has been on.

Some other thoughts:

One good way for Mylar to share data with ComicTagger is to write out a pickled python object, with both apps using a shared python class.

And yes, you do want to convert to ZIP if you want to CBL to read in the tags from the archive comment. However, as you might have seen in the app, the RAR comment can hold CBI very nicely, and it would be great if CBL would support the reading of this. I have no idea if development is still going on there, but it would be cool if you can add your voice to this thread:

http://forums.comicbooklover.com/discussion/51/comicbookinfo-in-rar-comments

I would say to experiment with this a bit, since you probably have a working Mylar set up. Write up a bash script that bet set up as a "Post-Processing Script" in sabnzbd. It should verify that it's a comic, do your rar-to-zip conversion, and then run comictagger with it's own online search. Once you have that working, we can figure out how to engage with the Mylar developer.
avatar
ComicTagger
Admin

Posts : 208
Join date : 2012-12-02

View user profile http://comictagger.forumotion.com

Back to top Go down

Re: Comictagger and Mylar

Post  ComicTagger on Tue Dec 11, 2012 8:49 pm

Forgot to mention, you can test if a file is a rar or not just by running the "file" command, standard on all Unix systems.

avatar
ComicTagger
Admin

Posts : 208
Join date : 2012-12-02

View user profile http://comictagger.forumotion.com

Back to top Go down

Re: Comictagger and Mylar

Post  manders2600 on Wed Dec 12, 2012 8:17 am

Well, I should let you know ahead of time that my bash scripting skills are horrible. I am sure there is an easier and more elegant way to do this, but here is something, anyway.

The wiki for sabnzbd userscripts is offline at the moment, so I'm not sure exactly which variables they set already (if I recall they make some things easier). No doubt I'll have to change something in here, but this would be one way to do it straight-up.

I am still using malor89's perl script for checking file types and changing extensions simply because it is easier than rewriting the whole thing in bash, lol. I'll probably go ahead and do that a bit later, but I want to make sure I've got the Sabnzbd specifics correct before I go any further, and to do that I'll need the sabnzbd wiki to come online.

Code:
#!/bin/bash

##    IMPORTANT    ##
## Set the directory sabnzbd downloads and extracts your comics to ##
## You will need to put checktype.pl from malor89's perl scripts in your "downloadpath" directory ##


downloadpath=/your/download/path/


## Sets up directories ##
mkdir "$downloadpath"temp/
comicpath="$downloadpath"temp/
cp "$downloadpath"checktype.pl "$comicpath"
cd "$comicpath"

## Takes all .cbr and .cbz files and dumps them to processing directory ##
for file in "$downloadpath"*.cbr "$downloadpath"*.cbz; do
mv "$file" "$comicpath"
done

## Changes filetype extensions when needed (could be scripted in bash, but quicker this way). ##
perl checktype.pl
rm renamed.txt

## Changes any cbr files to cbz files for insertion of metadata ##
for file in "$comicpath"*.cbr; do
mv "$file" "${file%.cbr}.rar"
done
for file in "$comicpath"*.rar; do
unrar x "$file"
rm "$file"
done
for D in ./* ; do
zip -rm "$D.zip" "$D" -x "*.cbz" "*.pl"
done
for file in "$comicpath"*.zip; do
mv "$file" "${file%.zip}.cbz"
done

## Tag and clean up ##
for file in "$comicpath"*.cbz; do
./comictagger.py -s -t cr -f -o  "$file"
mv "$file" "$downloadpath"
done
rm -r "$comicpath"

manders2600

Posts : 12
Join date : 2012-12-11

View user profile

Back to top Go down

Re: Comictagger and Mylar

Post  ComicTagger on Wed Dec 12, 2012 4:09 pm

Well, at first glance, it sure looks good to me.

Some comments with respect to comictagger:

  • Your call to "./comictagger.py" expects the ComicTagger to be in the working directory of the post-processing script, and that could fail unless you've got it there in "$comicpath. Usually best to give the full path of your tool, or expect in your path ($PATH).

  • Since you're running the python source version, you will need the Python Imaging Library installed for the online search to work properly. Maybe you have this already? You can test by runnning python, and then just the one line: "import Image". If that complains with an ImportError, you don't have it. If you're on Mac, it might be easier to use the binary package, and run the "ComicTagger" program in the app bundle, instead, so you don't have to work about dependencies.

avatar
ComicTagger
Admin

Posts : 208
Join date : 2012-12-02

View user profile http://comictagger.forumotion.com

Back to top Go down

Re: Comictagger and Mylar

Post  manders2600 on Wed Dec 12, 2012 6:08 pm

ComicTagger wrote:Well, at first glance, it sure looks good to me.

Thanks!

There are a few issues, though, as I am able to check the wiki page for sabnzbd (back up and running, hooray).

1. Sabnzbd already assigns a variable ($1) to the full path of the downloaded and extracted files, which actually works a lot better, but I'll have to change the variable in the script.
2. Need to insert return codes on success or failure.
3. Need to call mylar post-processing script following success.
4. User will need to input path to script directory (same directory as this script) unless:
Is there any way a bash script can be aware of its own directory, and then use this as a path to call another script? I was hoping one of the variables sent by sabnzbd would have been the configured path of the post-processing script folder, but no dice.
5. Need to advise user to put checktype.pl in script directory, then have it copy from there, as it will not be possible to put in ($1).
6. Need to remove checktype.pl altogether and write this in bash


ComicTagger wrote:Some comments with respect to comictagger:

  • Your call to "./comictagger.py" expects the ComicTagger to be in the working directory of the post-processing script, and that could fail unless you've got it there in "$comicpath. Usually best to give the full path of your tool, or expect in your path ($PATH).

  • Since you're running the python source version, you will need the Python Imaging Library installed for the online search to work properly. Maybe you have this already? You can test by runnning python, and then just the one line: "import Image". If that complains with an ImportError, you don't have it. If you're on Mac, it might be easier to use the binary package, and run the "ComicTagger" program in the app bundle, instead, so you don't have to work about dependencies.


Yep, the first one is just me being retarded late at night. I was cleaning up the script (sommenting, changing paths) as I had been using absolute paths to test, and I messed that one up. My feeling is that I would have it be set up as an user variable.

Yep, I have the Python Imaging Library (I'm running Linux Mint). For anyone else using an Ubuntu-based distro, the Python Imaging Library dependency is automatically installed when adding "python" from the repo.


Last edited by manders2600 on Thu Dec 13, 2012 8:22 am; edited 3 times in total

manders2600

Posts : 12
Join date : 2012-12-11

View user profile

Back to top Go down

Re: Comictagger and Mylar

Post  manders2600 on Wed Dec 12, 2012 7:41 pm

Could actually use some help, I'm not sure what I'm doing wrong with this checktype script. I've tested it with mislabeled cbr and cbz files, but while it correctly recognizes the file type, it does not do the rename, but rather gives the output that everything is fine.

Am I missing something here?


EDIT: Nevermind, I was missing something really simple. The below code should work:

bash file:
Code:
#!/bin/bash

comicpath=/path/to/comics/

for file in "$comicpath"*cbr; do
   filetype=`file "$file"`
   if [[ "$filetype" == *Zip* ]]
   then mv "$file" "${file%.cbr}.cbz"
   else echo "cbrs are fine"
   fi
done

for file in "$comicpath"*cbz; do
   filetype=`file "$file"`
   if [[ "$filetype" = *RAR* ]]
   then mv "$file" "${file%.cbz}.cbr"
   else echo "cbzs are fine"
   fi
done

manders2600

Posts : 12
Join date : 2012-12-11

View user profile

Back to top Go down

Re: Comictagger and Mylar

Post  manders2600 on Thu Dec 13, 2012 8:21 am

Okay, here is the bash script, completed and functional, except for the part about calling the mylar post-processing.

I have three questions, since as I said, I really don't know python at all:

1. Is there a way to pass variables from this bash script to the python script without editing the python script at all? I don't want to alter the mylar post-processing if at all possible, but I need to pass the variables supplied by sabnzbd (original nzb name, etc) in order for it to work.

2. Is there an easy way to convert this bash script to python? Currently, this script works in linux, but it would be much more compatible (and cleaner!) if it were in python.

3. Is there a specific list of variables that can be used to rename files, or is it just all of the tagging variables scraped from comicvine? If the latter, is there a quick reference for that somewhere?

Thanks!

Code:
#!/bin/bash

##    IMPORTANT    ##
##  YOU MUST PLACE THIS SCRIPT IN YOUR SABNZBD SCRIPTS FOLDER ##
##  AND YOU MUST SET THE COMICTAGGER DIRECTORY/PATH BELOW ##
##  FINALLY, YOU MUST MAKE THIS FILE EXECUTABLE (sudo chmod a+x /path/to/cmtagmylar.sh) ##


## Set the directory in which comictagger.py is located - IMPORTANT - ##
comictaggerpath=/path/to/comictagger/



## Sets up other directories ##
downloadpath="$1"/
sabnzbdscriptpath=$(dirname $0)
mkdir "$downloadpath"temp/
comicpath="$downloadpath"temp/
cd "$comicpath"

## Takes all .cbr and .cbz files and dumps them to processing directory ##
for file in "$downloadpath"*.cbr "$downloadpath"*.cbz; do
mv "$file" "$comicpath"
done

## Changes filetype extensions when needed ##
for file in "$comicpath"*cbr; do
filetype=$(file "$file")
if [[ "$filetype" == *Zip* ]]
then mv "$file" "${file%.cbr}.cbz"
else echo "cbr is good to go"
fi
done
for file in "$comicpath"*cbz; do
filetype=$(file "$file")
if [[ "$filetype" == *RAR* ]]
then mv "$file" "${file%.cbz}.cbr"
else echo "cbz is good to go"
fi
done

## Changes any cbr files to cbz files for insertion of metadata ##
for file in "$comicpath"*.cbr; do
mv "$file" "${file%.cbr}.rar"
done
for file in "$comicpath"*.rar; do
convertrarname="$file"
unrar x "$file"
rm "$file"
done
for D in ./* ; do
zip -rm "$convertrarname.zip" "$D" -x "*.cbz" "*.pl"
done
for file in "$comicpath"*.zip; do
mv "$file" "${file%.rar.zip}.cbz"
done

## Clean up temp directory and move files back to original directory ##
for file in "$comicpath"*.cbz; do
python "$comictaggerpath"comictagger.py -s -t cr -f -o  "$file"
mv "$file" "$downloadpath"
done
cd "$sabnzbdscriptpath"
rm -r "$comicpath"


## Will Run Mylar Post=processing In Future ##

manders2600

Posts : 12
Join date : 2012-12-11

View user profile

Back to top Go down

Re: Comictagger and Mylar

Post  Sponsored content


Sponsored content


Back to top Go down

View previous topic View next topic Back to top


 
Permissions in this forum:
You cannot reply to topics in this forum