Author Topic: SVN -> Git Conversions  (Read 117 times)

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 3960
SVN -> Git Conversions
« on: September 14, 2017, 05:18:57 AM »
I noticed some of the initial imports of projects into Git still maintain the SVN trunk/tags/branches structure. I read a couple articles today that deal with converting those to Git style branches and tags.

Migrating From SVN to Git Preserving Branches And Tags

Convert Git-SVN Tag Branches to Real Tags

It comes down to using extra flags during the git svn init, and then possibly some post conversion of git remote branches into tags. I tried it out using the NetFixServer project. This was possible even though the project has been deleted from the head revision in SVN, since git-svn will search through history and find it in previous revisions. Using the --stdlayout flag removes the extra SVN folder structure, and converts the contents into the expected Git branches:

Code: [Select]
mkdir NetFixServer
cd NetFixServer
git svn init https://svn.outpostuniverse.org:8443/svn/outpost2/NetFixServer/ --stdlayout
git svn fetch

Alternatively, the "--stdlayout" flag can be replaced and expanded into "-t tags -b branches -T trunk". This lets you adjust for any naming differences to the standard layout (case sensitive).


There was also the bit about converting the remote branches to proper Git tags. I modified one of the examples from the above links to the following:
Code: [Select]
git for-each-ref refs/remotes/origin/tags | cut -d / -f 5- | while read ref;
    do git tag -a "$ref" -m"say farewell to SVN" "refs/remotes/origin/tags/$ref"
done

I suspect we can probably tag a bit better, but that's a start. The folder structure is much cleaner this way. What I don't like about the tagging is the extra commit message. It looks a bit like a double commit, once from the SVN tagging, and once from the Git tagging. Perhaps the original revision can be tagged directly with the SVN tag message.

Offline leeor_net

  • Administrator
  • Hero Member
  • *****
  • Posts: 1629
    • LairWorks Entertainment
Re: SVN -> Git Conversions
« Reply #1 on: September 14, 2017, 10:26:32 AM »
These are the commands that I used for it:

Code: [Select]
git svn clone URL_HERE
git filter-branch -f --tree-filter "rm -fr TestFiles" --prune-empty HEAD
git remote add origin URL_TO_GITHUB_PROJECT
git remote -v
git push origin master

Though it seems like yours are more about converting the 'standard layout' of SVN repositories into a regular layout for Git.
- Leeor
LairWorks Entertainment

Titanum UFO's

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 3960
Re: SVN -> Git Conversions
« Reply #2 on: September 15, 2017, 03:31:23 AM »
Ahh, ok. Are you cloning the entire SVN repository, and then filtering from there to extract a project? Or by "URL_HERE", do you mean the URL to the project subfolder within the SVN?

I already have a full git-svn checkout of the entire SVN repo, so it might be faster to just filter out a project. I'm not sure how to combine that with the folder layout adjustment though.

I didn't cover the part about pushing to repo up to GitHub, which seems to be your last 3 commands.

Offline leeor_net

  • Administrator
  • Hero Member
  • *****
  • Posts: 1629
    • LairWorks Entertainment
Re: SVN -> Git Conversions
« Reply #3 on: September 15, 2017, 03:40:13 AM »
Right, URL_HERE is just the SVN repository's URL. Git clones the SVN with full history into a local directory. I then apply a filter based on file/directory (in some cases have to do it several times... there's probably a better way to do it but this works). I then compress it to filter out any revision history that is now empty, then attach the local repository to a remote repository and push/sync it with remote master.

Your command set looks to rework the directory structure. That could be done after the clone and either before or after filtering out files.
- Leeor
LairWorks Entertainment

Titanum UFO's

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 3960
Re: SVN -> Git Conversions
« Reply #4 on: September 29, 2017, 10:50:10 PM »
Vagabond was asking about putting OP2MapImager up onto GitHub. I thought I'd document the process.

The OP2MapImager project did not have the standard SVN trunk/tags/branches folder structure, so no special conversion was needed there. OP2MapImager had the standard format OP2 tilesets as part of the project. Since these were proprietary, they needed to be filtered out. This removes the files not just from the HEAD revision, but also from all previous commits, ensuring there is no trace left in the repository. The project also previously had some maps for test data. The test data was removed by a previous commit, but the map files still existed in history. Again, since they were proprietary, they needed to be filtered out. The project also contained 7za.exe and the FreeImage library. Checking the licensing files, it appears they both allow for redistribution, and so they were left in with no changes.

The conversion steps were:
Code: [Select]
mkdir OP2MapImager
cd OP2MapImager/
git svn init https://svn.outpostuniverse.org:8443/svn/outpost2/GameResources/OP2MapImager/
git svn fetch
git filter-branch --tree-filter "rm -f OP2MapImager/*.bmp" --prune-empty HEAD
git filter-branch -f --tree-filter "rm -rf OP2MapImager/TestFiles/" --prune-empty HEAD

To upload the results to GitHub, a GitHub repository needs to be created first. This can be done from the GitHub website. To resulting URL was: https://github.com/OutpostUniverse/OP2MapImager.git

To upload the contents of the local repository to the new empty GitHub repository:
Code: [Select]
git remote add origin https://github.com/OutpostUniverse/OP2MapImager.git
git push -u origin master


The project is now up on GitHub. The URL for the page is:
https://github.com/OutpostUniverse/OP2MapImager

« Last Edit: September 29, 2017, 10:52:57 PM by Hooman »

Offline Vagabond

  • Sr. Member
  • ****
  • Posts: 449
Re: SVN -> Git Conversions
« Reply #5 on: September 30, 2017, 02:15:25 AM »
Hooman, thanks for uploading OP2Imager to Github. I just downloaded it, compiled it, and created the gitignore file.

We will have to decide how to deal with the tilesets missing from the source code on GitHub. Ideally it would be contained in the game download, but I'll try hosting it on the Wiki temporarily. The ReadMe can include where to fetch them. For the next release of Outpost 2, we should consider including the non Outpost 2 specific tileset images since the game runs fine with generic BMPs. This way the ReadMe can just focus on telling the user to get them from the Outpost 2 download. Probably worth discussing in more detail in a different thread if needed to keep from hijacking the purpose of this one.

I'll also look at collapsing the project and solution file into the same directory so the ReadMe file exists in the root directory instead of having to drill down a directory. This will make for a nicer experience browsing the code on GitHub.

-Brett

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 3960
Re: SVN -> Git Conversions
« Reply #6 on: October 01, 2017, 12:22:51 AM »
We should also look into using Git submodules. That's something I avoided for now, just to get the project up there. Ideally, the OP2Utility project should be added as a submodule to both the OP2MapImager project, and the OP2Archive project. Since submodules are tied to a commit and only updated manually, that also means you can update the OP2Utility project without breaking the other projects. Right now, it's left to the developer to clone both projects individually, and set appropriate paths for building.

The tilesets should probably be zipped and posted to the forum somewhere. Good idea to repack them for the next release. Might also be incentive to get the native OP2 format bitmap loading code done, though I'm still in favour of repacking.

I agree with collapsing the project structure and putting the ReadMe in the root folder. Having the ReadMe.md file appear on the repo page does improve the GitHub experience.
« Last Edit: October 01, 2017, 01:06:24 AM by Hooman »