This is the old SliTaz forum - Please use the main forum.slitaz.org

Little Bash script - need help
  • molmol September 2010

    Hi,

    I would like to copy my pictures to a specified folder (home in the exemple)

    The script I have here, can manage sub-folders and even with space in their name

    cd /media/Iomega/
    find . -type f -name "*.jpg" -exec cp {} /home \ ;

    The problem is that when a file has same name than a picture (previously copied in /home/ ) , it replace it

    I knoy that there is the cp -i option, but I want to have automatic rename procedure

    Thx
  • ChristopheChristophe September 2010
    a way to do that is to execute an external script that will test if the file exists and if so, will append a random string to the filename. In the past i have used the PID of the process to do that, this creates some random number which is usually random enough to avoid collision with an existing file name but you can also test for the existence of FILENAME.Number.jpg where number starts at 1 and increment number until there is no collision.

    On another note i may be wrong but i seem to remember that the -exec option is not implemented in that version of slitaz ? May be wrong, as I said.
  • molmol September 2010
    Hi Christophe

    Mhh , seems very complex your method. But , yes, my script above with "-exec" works on Slitaz .

    I was wondering if a loop like

    -For each
    ----If exist
    -------rename automaticaly
    -------cp
    ----else
    -------cp
    -done

    could be ok. But I 'n mot able to browse sub-directories (moreover if there's is space in their name)

    Anyone can help me ?
  • ChristopheChristophe September 2010
    what you are doing is basically what i was suggesting
    to browse subdirectroy you need to use the back quotes

    eg

    for i in `find . -type f -print`
    do
    echo $i
    done

    you will see all subfiles
    now just replace echo $i with your code for each individual $i
    i do not think the space in the name will be an issue here

  • molmol September 2010
    Ok , here's the script . It even check if the size of 2 files with same name are equal or not (just to know if same picture) and add a "0"

    But it cannot go into sub-directories with name with space

    :(



    cd /media/Iomega/
    for i in `find . -type f -name "*.jpg" -print`
    do
    --if [ -f /home/$i ]
    ----then
    ----echo SAME NAME
    ----j="/home/$i"
    ----filesize_new=$(stat -c%s "$i")
    ----filesize_old=$(stat -c%s "$j")
    ----if [ $filesize_new == $filesize_old ]
    -------then
    -------echo AND SAME SIZE
    -------else
    -------echo BUT NOT SAME PICTURE
    -------mv "$j" "$j"0
    -------cp "$i" /home/
    ----fi
    --else
    ----echo OK YOU CAN DIRECT COPY
    ----cp $i /home/
    --fi
    done



    Please help, I spend lots of time
  • GokhlayehGokhlayeh September 2010
    You need to systematicaly quote variables if you want to have space support, like :

    cd /media/Iomega/
    for i in `find . -type f -name "*.jpg" -print`
    do
    --if [ -f "/home/$i" ] # QUOTE HERE
    ----then
    ----echo SAME NAME
    ----j="/home/$i"
    ----filesize_new=$(stat -c%s "$i")
    ----filesize_old=$(stat -c%s "$j")
    ----if [ $filesize_new == $filesize_old ]
    -------then
    -------echo AND SAME SIZE
    -------else
    -------echo BUT NOT SAME PICTURE
    -------mv "$j" "$j"0
    -------cp "$i" /home/
    ----fi
    --else
    ----echo OK YOU CAN DIRECT COPY
    ----cp "$i" /home/ # QUOTE HERE
    --fi
    done
  • babaorumbabaorum September 2010
    Hi, here is a slightly enhanced version of the same script.
    SOURCE="/media/Iomega"
    DEST="/home"
    echo_date_do() {
    cmd=$*
    eval "$cmd" && echo "$(date) : $cmd"
    }
    [ -n "$1" -a -n "$2" ] && SOURCE="$1" && DEST="$2"
    if [ -r "$SOURCE" -a -d "$SOURCE" -a -r "$DEST" -a -d "$DEST" ]; then
    for file in $(find "$SOURCE" -type f -name "*.jpg" -print); do
    bn=$(basename "$file")
    if [ -f "${DEST}/$bn" -a $(stat -c%s "$file") -eq $(stat -c%s "${DEST}/$bn") ]; then
    echo_date_do 'mv "${DEST}/$bn" "${DEST}/${bn}0"'
    fi
    echo_date_do 'cp "$file" "$DEST"'
    done
    exit 0
    else
    exit 1
    fi

    It lets you set alternative source and destination directories as arguments at command line, if needed (default ones are those you usually need, as shown in your request) ; the "copy" and "move" actions are all simply logged "as is" in standard output.
    I've removed a lot of intermediate lines, to get the essential ones.
    Finally, the script tests validity of both source and destination directories and returns an exit value, may be interesting for later logging/debuging or to re-use this script as wrapped in more complex scripts ... ?

    As an example, you should be able to log all the actions done by the script with such a launcher (another script, shell command line...) :
    (let's say that you name your backup script "imgbkup")
    imgbkup >"/home/logs/imgbkup/$(date -I'seconds')-imgbkup.log"
    You would get a whole directory (/home/logs/imgbkup) filled with successive logfiles from the backup script.

    Regards,
    --Babaorum
  • molmol September 2010
    omg :)

    Not sure I may understand all little tricks you add (I'm illiterate with Bash)

    But there's another thing you can enhance . Look, a "0" is add to the name if a
    file has same name in /home/ directory.

    So you have something like "picture.jpg0" .... :/

    Maybe change the letter before ".jpg"

    Also . "jpeg" , "png" "gif" "bmp" "avi" (actually all medias) may be added .

    Another limit. What's happend when a file ALREADY has the name "picture.jpg0" ? This time it's erase ! So rather adding a "0" , maybe adding something unique is better . (Like a some characters related with the size of the file)

    __________________


    The idea could be to create a tiny Slitaz tool , kind of Data Manager with gtk frontend which be able to import, classify and rename all sort of files remaining in disorder on a large HD . All photos with same names but different folders forgotten in the abyss of an old Windows partitions. (A bit like XnView, but far more light)

    _______Isn't a good idea ?__________________


    Thx 4 the script
  • molmol September 2010
    @Gokhlayeh

    Your fix does'nt work with sub-folders with space

    @babaorum

    Your scipt doesn't work at all :( Did you test it ?
  • molmol September 2010
    This work, but answer me each time if I want to replace the file in /home (if same name)

    cd /media/Iomega/
    find . -type f -name "*.jpg" -exec cp {} /home \;

    Is there anyway to automaticly say "no" but, at least, copy with another name ?
  • GokhlayehGokhlayeh September 2010
    @mol : the problem is now the for .. in .. ; do .. ; done. when there's space the for loop split answer in several pieces. for can't read an answer line by line. To do that we must use a `func` | while read line; do .. ; done. So this works :
    find . -type f -name "*.jpg" -print | while read i; do
    if [ -f "/home/$i" ]; then
    echo SAME NAME
    j="/home/$i"
    filesize_new=$(stat -c%s "$i")
    filesize_old=$(stat -c%s "$j")
    if [ $filesize_new == $filesize_old ]; then
    echo AND SAME SIZE
    else
    echo echo BUT NOT SAME PICTURE
    mv "$j" "$j"0
    cp "$i" /home/
    fi
    else
    echo OK YOU CAN DIRECT COPY
    cp "$i" /home/
    fi
    done
  • babaorumbabaorum September 2010
    Well, I've tested my script but I'm used to never putting any space in my filenames, it may have confused myself.
    The '0' is mispositionned indeed and would be better replaced by a random string. Here are the bug fixes, mixing Gokhlayeh's and my scripts.
    SOURCE="/media/Iomega"
    DEST="/home"
    case "$1" in
    --help)
    echo 'Usage : '$(basename "$0")'[--overwrite] [source-directory destination-directory]' && exit 0
    break;;
    --overwrite) OVERWRITE='true'; shift; break;;
    *) OVERWRITE='false';;
    esac
    [ -n "$1" -a -n "$2" ] && SOURCE="$1" && DEST="$2"
    if [ -r "$SOURCE" -a -d "$SOURCE" -a -r "$DEST" -a -d "$DEST" ]; then
    find "$SOURCE" -type f -name "*.jpg" -print | while read $file; do
    bn=$(basename "$file")
    if [ -f "${DEST}/$bn" -a $(stat -c%s "$file") -eq $(stat -c%s "${DEST}/$bn") ]; then
    case $OVERWRITE in
    true) rm -f ${DEST}/$bn;;
    false) mv "${DEST}/$bn" "${DEST}/${bn%%.jpg}_$(cat /dev/urandom | tr -dc A-Za-z0-9 | head -c 8).jpg";;
    esac
    fi
    cp "$file" "$DEST"
    done
    exit 0
    else
    exit 1
    fi


    Another way to add a non-random but still unique stamp is to use the date and time : YYYYMMDDHHMMSS (but unless you wanna risk to get same time stamp for same file within the same minute, you must get seconds too and get a 14-characters timestamp... maybe a little long if you plan to burn a CD w/ ISO/Joliet filename length limitation ?).
    Here is the code for unique timestamp:
    date '+%Y%m%e-%H%M%S'
    instead of
    cat /dev/urandom | tr -dc A-Za-z0-9 | head -c 8
    (8 indicates the number of characters, it could be more or less)

    I'll pay attention to your feedback if any :)
  • GokhlayehGokhlayeh September 2010
    @babaorum : The good syntax is not "while read $file" but "while read file". mol is right, you should test your code :D.

    Here :
            if [ -f "${DEST}/$bn" -a $(stat -c%s "$file") -eq $(stat -c%s "${DEST}/$bn") ]; then
    case $OVERWRITE in
    true) rm -f ${DEST}/$bn;;
    false) mv "${DEST}/$bn" "${DEST}/${bn%%.jpg}_$(cat /dev/urandom | tr -dc A-Za-z0-9 | head -c 8).jpg";;
    esac
    fi

    This loop run only if "$file" and "${DEST}/$bn" are the same, right ? Shouldn't be the opposite ? like :
    if [ -f "${DEST}/$bn" ] && [ $(stat -c%s "$file") -ne $(stat -c%s "${DEST}/$bn") ]; then

    Then why not use the block count as suffix ? This way if you run the script several times it copy the file with suffix only one time. Note : in this case, if a file with same name exists in destination and is not the same that the one we what to copy, the file is copied with another name. In babaorum script, the file in destination is moved then the file we want to copy is copied with the same name.

    Script with theses changes + search pattern support + if you use overwritte after a normal use, move the renamed & copied files at right place :
    #!/bin/sh

    # Default values.
    SOURCE="/media/Iomega"
    DEST="/home"
    SEARCH="*.jpg *.png"
    OVERWRITE='false'

    # Print help if requested and look for arguments in command line.
    for i in $(seq 1 2); do
    case "$1" in
    --help)
    echo 'Usage : '$(basename "$0")'[--overwrite] [--search "pattern1 pattern2" ] [source-directory destination-directory]'
    exit 0
    ;;
    --overwrite)
    OVERWRITE='true' && shift
    ;;
    --search)
    SEARCH="$2" && shift 2
    ;;
    *) ;;
    esac
    done
    [ -n "$1" -a -n "$2" ] && SOURCE="$1" && DEST="$2"

    # Execute the script.
    if [ -r "$SOURCE" -a -d "$SOURCE" -a -r "$DEST" -a -d "$DEST" ]; then

    # Set -f disable pathname expansion (globbing).
    set -f
    if [ "${SEARCH%% *}" = "$SEARCH" ]; then
    command='find $SOURCE -name "'$SEARCH'"'
    else
    command='find $SOURCE -name "'${SEARCH%% *}'"'$(for i in ${SEARCH#* }; do echo -n " -o -name \"$i\""; done)
    fi
    set +f

    # Run the main loop.
    eval "$command" | while read file; do
    bn=$(basename "$file")
    bc=$(stat -c%s "$file")
    if [ -f "${DEST}/$bn" ] && [ "$bc" -ne $(stat -c%s "${DEST}/$bn") ]; then
    case $OVERWRITE in
    true)
    echo "Removing ${DEST}/$bn"
    rm -f "${DEST}/$bn"
    if [ -f "${DEST}/$bn.$bc" ]; then
    echo "Moving ${DEST}/$bn.$bc to ${DEST}/$bn"
    mv "${DEST}/$bn.$bc" "${DEST}/$bn"
    fi
    ;;
    false)
    bn="$bn.$bc"
    ;;
    esac
    fi
    if [ -f "${DEST}/$bn" ]; then
    echo "File ${DEST}/$bn already exists."
    else
    echo "Copying $file to $DEST/$bn"
    cp -a "$file" "$DEST/$bn"
    fi
    done
    exit 0
    else
    echo "Error : Can't find $SOURCE or $DEST"
    exit 1
    fi


    Hope all run fine. If you don't understand something in this code, just ask or add some echo to check variables and functions returns.
  • molmol September 2010
    VEry nice ! But I can't understnd all that stuff

    Last detail : It's ok if the SOURCE has no space "/media/Iomega" ...
    but if the SOURCE has something like "/media/Iomega HDD" , it doesn't work
  • GokhlayehGokhlayeh October 2010
    @mol : you're right, I forgot quotes for $SOURCE in find command. Please note that when giving argument with spaces in command line or in variable you need to quote it. i.e :
    scriptname /media/Iomega HDD dest
    this doesn't works
    scriptname "/media/Iomega HDD" dest
    this works.

    Fixed script:
    #!/bin/sh

    # Default values.
    SOURCE="/media/Iomega"
    DEST="/home"
    SEARCH="*.jpg *.png"
    OVERWRITE='false'

    # Print help if requested and look for arguments in command line.
    for i in $(seq 1 2); do
    case "$1" in
    --help)
    echo 'Usage : '$(basename "$0")'[--overwrite] [--search "pattern1 pattern2" ] [source-directory destination-directory]'
    exit 0
    ;;
    --overwrite)
    OVERWRITE='true' && shift
    ;;
    --search)
    SEARCH="$2" && shift 2
    ;;
    *) ;;
    esac
    done
    [ -n "$1" -a -n "$2" ] && SOURCE="$1" && DEST="$2"

    # Execute the script.
    if [ -r "$SOURCE" -a -d "$SOURCE" -a -r "$DEST" -a -d "$DEST" ]; then

    # Set -f disable pathname expansion (globbing).
    set -f
    if [ "${SEARCH%% *}" = "$SEARCH" ]; then
    command="find \"$SOURCE\" -name \"$SEARCH\""
    else
    command="find \"$SOURCE\" -name \"${SEARCH%% *}\""$(for i in ${SEARCH#* }; do echo -n " -o -name \"$i\""; done)
    fi
    set +f

    # Run the main loop.
    eval "$command" | while read file; do
    bn=$(basename "$file")
    bc=$(stat -c%s "$file")
    if [ -f "${DEST}/$bn" ] && [ "$bc" -ne $(stat -c%s "${DEST}/$bn") ]; then
    case $OVERWRITE in
    true)
    echo "Removing ${DEST}/$bn"
    rm -f "${DEST}/$bn"
    if [ -f "${DEST}/$bn.$bc" ]; then
    echo "Moving ${DEST}/$bn.$bc to ${DEST}/$bn"
    mv "${DEST}/$bn.$bc" "${DEST}/$bn"
    fi
    ;;
    false)
    bn="$bn.$bc"
    ;;
    esac
    fi
    if [ -f "${DEST}/$bn" ]; then
    echo "File ${DEST}/$bn already exists."
    else
    echo "Copying $file to $DEST/$bn"
    cp -a "$file" "$DEST/$bn"
    fi
    done
    exit 0
    else
    echo "Error : Can't find $SOURCE or $DEST"
    exit 1
    fi


    You can ask for info if you want to understand the code. Here's how it works :
    * First define default values in case there's not arguments in command line
    * Then looking for options (--*) in command line twice (seq 1 2), because --overwrite and --search can be given in the same command line. Once argument is parsed, erase it with shift.
    * After this loop, if two arguments are remaining, use them as source and dest
    * Create and store the find command in the variable command
    * In the begin of main loop, 'eval "$command"' execute the find command stored in the variable and send each answer to the main loop.
    * In main loop each action is preceded by an echo command wich explain what it does. variable bn is the name of file; variable bc is the bloc count.

    Hope this one doesn't have issues.
  • ChristopheChristophe October 2010
    Not as sophisticated than yours, but will work to copy all files in ./. provided there is no existing files of the extension we are looking for in ./.

    for EXT in echo jpg jpeg ; do find . -type f -name *$EXT -print | awk -v QQ='"' -v xx=$EXT ' { system( "cp " QQ $0 QQ " `basename " QQ $0 QQ " " xx "`" rand()*1000000 "." xx ) } ' ; done

    I will admit i tried some alternatives before having this one working as expected. Will easily be adjusted to add other extension or a specific path where to copy the files. All files get appended a 6 numbers random number before copying, which should be good enough (but there is still a non zero chance of collision)
  • babaorumbabaorum October 2010
    Hi Gokhlayeh,

    It was not an error from me, my test stated a different action than yours stated.

    Mine was the following :
    Test = "file already exists and seems to be the same (=same filesize)"
    - if user wants to overwrite : erase the existing file and let the "cp" do its jobs later in code
    - if not, move the existing file while renaming it
    ...
    and then, out of the test : copy files as wanted as main action

    Yours was different. Both are correct, depending of what mol wants indeed : get a backup of "apparently" same image files, or get a backup of "apparently" different image files.
    Both usages have their interest indeed, I had planned on one while you had planned on the other. :-)

    while read $file was a typing error, you should have guessed. I changed the lining of code while editing the message, as often concerning myself... but I insist : I *DO* test my code.
    While I am neither a Linux pro nor a programmer at all, I can make errors, I'm sorry. But I also tried to bring some new ideas/functionalities ... :-S
  • babaorumbabaorum October 2010
    I haven't understood this, can you explain please :
        # Set -f disable pathname expansion (globbing).
    set -f
    if [ "${SEARCH%% *}" = "$SEARCH" ]; then
    command="find \"$SOURCE\" -name \"$SEARCH\""
    else
    command="find \"$SOURCE\" -name \"${SEARCH%% *}\""$(for i in ${SEARCH#* }; do echo -n " -o -name \"$i\""; done)
    fi
    set +f
  • babaorumbabaorum October 2010
    Hmmm I don't understand anything to be honest.
    - why looking for a "$bn.$bc" file to rename "$bn" in $DEST after removing existing image file when user wants to overwrite ?
    - in case of file already existing in $DEST and user not wanting to overwrite, why not backing up the existing file in $DEST as a renamed file in same dir, rather than adding a unique stamp to the file to-be-copied ?
    - while adding this stamp, won't it figure at the very end of filename, past the extension file ? If so, wouln't it be better to place it before the extension, like mol make us notice previously ?

    May be is it a problem of difference of logic, I'ms orry, but I can read your code and read again, I cannot understand. :-((
  • GokhlayehGokhlayeh October 2010
    @babaorum :
    While I am neither a Linux pro nor a programmer at all, I can make errors, I'm sorry. But I also tried to bring some new ideas/functionalities ... :-S
    It was not an attack, just a joke. I'm sorry about that if you take it another way. I'm new to bash and I make a lot of errors ;)

    About the part of code you copied :
    set -f/set +f disable/enable pathname expansion. If you don't understand this : open your terminal in your home and type : set -f; echo *; set+f echo *
    I do that because I don't want the wild card be interpreted when setting the command variable.
    So the loop set the command variable, which is the command used to find requested files. if [ "${SEARCH%% *}" = "$SEARCH" ] check if there's one or several input in the SEARCH variable. ${SEARCH%% *} is $SEARCH without the first space and all next text. If there's only one input command will be :
    find $SOURCE -name input
    else :
    find $SOURCE -name input1 -o -name input2 etc...
    ${SEARCH%% *} is input1; ${SEARCH#* } is $SEARCH without input1.

    - in case of file already existing in $DEST and user not wanting to overwrite, why not backing up the existing file in $DEST as a renamed file in same dir, rather than adding a unique stamp to the file to-be-copied ?
    I was thinking that the script shouldn't do anything with existing files if overwrite is not setted.
    - why looking for a "$bn.$bc" file to rename "$bn" in $DEST after removing existing image file when user wants to overwrite ?
    If you use the script without overwrite, the copied file is $bn.$bc in case a file with same name already existing. If you use it a second time with overwrite it move $bn.$bc to $bn (the real name). So the result is like you have used the script once with overwrite.

    - while adding this stamp, won't it figure at the very end of filename, past the extension file ? If so, wouln't it be better to place it before the extension, like mol make us notice previously ?
    Yes, it should be better :)

    As you said we write our script with different thinking about how it should does the work. So we have different results.

    Hope all is clear now.
  • babaorumbabaorum October 2010
    Hmm... variable substitutions do not work as I had wondered, I must get it back to my lessons :-S I have always seen them as integrated regexp but only the following form is using regexp: ${var_name/pattern/subst}... not the other ones !

    For the left part of the code, you say right : I do not have the same thinking pattern as you, so it is hard for me to say if it is good or bad (and I don't want to !). But it works, that makes the trick, isn't it ?

    I did not take your remarks in a bad way, I am not in such a bad mood, OK ? :-)
    I was just a little annoyed that you may think I do not take the time to test the code. But I'm a bad typer and it is precisely between my mind and my fingers that many errors happen. :-Z

    --Babaorum
  • babaorumbabaorum October 2010
    I know where I failed analyzing your var` substitutions : the usage of "set -f" before these var substs was confusing : I thought it also applied to var substs, but it isn't. If it was, the substitutions would not work as intended.
    So var substitutions do use regular expressions even in these forms. Whatever set +/-f is...
  • babaorumbabaorum October 2010
    thanks

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Apply for Membership

SliTaz Social