Today I got into quite some extensive bash programming for the first time. The reason was that I wanted to automate a task that has been bothering me for some days now: I had set up a new branch for a software project in SVN, and the project structure in the branch is quite different from the one in the trunk.
I want to keep the branch up to date with the trunk, but merging is not really an option, since I have all these tree conflicts now, and I can’t simply resolve them since people are still using the trunk. I expect the trunk to die soon anyway and the new structure to become the trunk. So I am doing something that is not really pretty: I am manually copying the changed trunk files to my branch (which means losing the svn commit messages from the original trunk commits, but I don’t intend to keep this up for more than a few days. I also make nice commit messages including the revision numbers and major changes in the trunk).
Needless to say, this is a dull task, and this was a great opportunity to get into some scripting. So what I ended up with is two scripts, one to pull the svn changes and log them, and another to copy the changed files to my beautiful new branch.
The first script is quite small, and looks like this:
#!/bin/bash
#do an svn up, delete the first line from the output (Revision xxx), cut the start of the lines (D, M...), sort the svn changes, and save the result
svn up | sed -e '/Revision/Id' | sort -k2 > changedfiles.txt
if [[ -s changedfiles.txt ]] ; then
copychanges.sh changedfiles.txt #call the actual copy script with the changes file
mv changedfiles.txt changedfiles.bak
else
echo "Everything is up to date."
fi
This does the “svn up” to get the new files, and pipes the output to sed. Sed deletes the line which says “updated to revision xyz”. After that, it sorts the lines by the filepaths ( using the sort parameter -k2 to sort only by the file paths in column 2, and not by the SVN flags “D,U,A”). Finally, the result is saved to file.
If the file is empty, nothing changed, and I am done. Otherwise, I call the other script (copychanges.sh), which will put the changed files into the corresponding folders in my branch. That script looks like this:
#!/bin/bash
#User provides a file that contains a list of filenames, one per line
#save current stdin
exec 6<&0
# make stdin the file that the user provided, and then read each line in the
# file into an array
exec < $1
numLines=`wc -l < $1`
for i in `seq 1 "$numLines"`
do
read filename
files["$i"]="$filename"
done
#Now that we're done loading the data into the array, restore stdin
exec 0<&6 6<&-
#iterate through the array, asking a question about each file
echo "Processing the following files:"
for file in "${files[@]}"
do
echo $file
done
echo
echo "What is the target directory?"
read targetProject #../datahub-maven
#ask for sourceRoot every time a file has a new prefix
sourceRoot=""
for file in "${files[@]}"
do
echo
status=${file:0:1} #'D' for deleted, 'U' for updated, 'A' for added
file=${file:5}
sourceRootLength=${#sourceRoot}
if [ $sourceRootLength -eq 0 ] || [ "${file:0:$sourceRootLength}" != "$sourceRoot" ]; then
sourceRoot=""
#let the source file be 'src/com/icon/db/util/Bla.java'
until [ -d "$sourceRoot" ]; do # ask until an existing directory has been specified
echo "What is the root for source file $file?"
read sourceRoot #src
if [ ${#sourceRoot} -eq 0 ]; then
break # file is in the root of the hierarchy
fi
done
sourceRootLength=${#sourceRoot}
echo "What is the corresponding target root for $sourceRoot?"
read targetRoot
until [ -d "${targetProject}/${targetRoot}" ]; do # ask until an existing directory has been specified
echo "What is the corresponding target root for $sourceRoot?"
read targetRoot #src/main/java
done
fi
#getting the extension part
fileLength=${#file}
extension=${file:$sourceRootLength:$fileLength} #/com/icon/db/util/Bla.java
echo "Extension: $extension"
targetLocation=${targetProject}/${targetRoot}${extension} #../datahub-maven/src/main/java/com/icon/db/util/Bla.java
#TODO take over the flags from svn, and warn if a modified file does not exist in target, or if an added file already exists
tdir=`exec dirname $targetLocation` # directory name without file
if [ $status == "U" ]; then
read -p "Copy $file to $targetLocation?" -n 1
if [[ $REPLY =~ ^[Yy]$ ]]; then
cp $file $targetLocation
else echo "Skipped $file"
fi
elif [ $status == "A" ]; then
read -p "Copy $file to $targetLocation?" -n 1
if [[ $REPLY =~ ^[Yy]$ ]]; then
if [ ! -d "`exec dirname $tdir`" ]; then
echo "Creating new directory $tdir"
mkdir -p "$tdir"
fi
cp $file $targetLocation
else echo "Skipped $file"
fi
elif [ $status == "D" ]; then
read -p "Delete $targetLocation?" -n 1
if [[ $REPLY =~ ^[Yy]$ ]]; then
rm $targetLocation
else echo "Skipped deletion of $targetLocation"
fi
fi
done
Quite a bit longer, but I really learned a lot here.
The problem I have is that I don’t know which files to put where in the new structure. For example, the old project might have a layout like src/java/dbutils/MyDbClass.java
The new project might need this class to be under src/main/java/dbutils/MyDbClass.java
So for each file path, the script asks me what the base path for the file is, and what the corresponding base path is in the branch. For example, the base path in the trunk here would be src, and the corresponding path in the branch is src/main/java. After that, the paths are the same. So the script prompts me once for every different path that appears in the svn log file (that’s why I sorted the output from svn in the other script), and copies the files to their new destination.
So basically you tell the script what the parent directory is in the old and new branch, and the script does the rest. I built in some verifications, like that the parent paths you specify must exist. Additionally, there is some fiddling around with the file names; I need to take them apart and put them back together a bit. But that’s basically it.
What I found out, being more of a java guy, is that it is really hard to remember when to use quotations and the like when comparing variables.
When comparing numbers, you use -eq. If you want to compare strings, you use == . Also, the syntax for finding out things about stings is not really intuitive. It’s short though, I have to admit that.
But all in all, it is not so hard to write some really nifty tools in bash.
