Author Topic: SVN -> Git Conversions  (Read 17304 times)

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
SVN -> Git Conversions
« on: September 14, 2017, 07:18:57 AM »
I noticed some of the initial imports of projects into Git still maintain the SVN trunk/tags/branches structure. I read a couple articles today that deal with converting those to Git style branches and tags.

Migrating From SVN to Git Preserving Branches And Tags

Convert Git-SVN Tag Branches to Real Tags

It comes down to using extra flags during the git svn init, and then possibly some post conversion of git remote branches into tags. I tried it out using the NetFixServer project. This was possible even though the project has been deleted from the head revision in SVN, since git-svn will search through history and find it in previous revisions. Using the --stdlayout flag removes the extra SVN folder structure, and converts the contents into the expected Git branches:

Code: [Select]
mkdir NetFixServer
cd NetFixServer
git svn init https://svn.outpostuniverse.org:8443/svn/outpost2/NetFixServer/ --stdlayout
git svn fetch

Alternatively, the "--stdlayout" flag can be replaced and expanded into "-t tags -b branches -T trunk". This lets you adjust for any naming differences to the standard layout (case sensitive).


There was also the bit about converting the remote branches to proper Git tags. I modified one of the examples from the above links to the following:
Code: [Select]
git for-each-ref refs/remotes/origin/tags | cut -d / -f 5- | while read ref;
    do git tag -a "$ref" -m"say farewell to SVN" "refs/remotes/origin/tags/$ref"
done

I suspect we can probably tag a bit better, but that's a start. The folder structure is much cleaner this way. What I don't like about the tagging is the extra commit message. It looks a bit like a double commit, once from the SVN tagging, and once from the Git tagging. Perhaps the original revision can be tagged directly with the SVN tag message.

Offline leeor_net

  • Administrator
  • Hero Member
  • *****
  • Posts: 2352
  • OPHD Lead Developer
    • LairWorks Entertainment
Re: SVN -> Git Conversions
« Reply #1 on: September 14, 2017, 12:26:32 PM »
These are the commands that I used for it:

Code: [Select]
git svn clone URL_HERE
git filter-branch -f --tree-filter "rm -fr TestFiles" --prune-empty HEAD
git remote add origin URL_TO_GITHUB_PROJECT
git remote -v
git push origin master

Though it seems like yours are more about converting the 'standard layout' of SVN repositories into a regular layout for Git.

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
Re: SVN -> Git Conversions
« Reply #2 on: September 15, 2017, 05:31:23 AM »
Ahh, ok. Are you cloning the entire SVN repository, and then filtering from there to extract a project? Or by "URL_HERE", do you mean the URL to the project subfolder within the SVN?

I already have a full git-svn checkout of the entire SVN repo, so it might be faster to just filter out a project. I'm not sure how to combine that with the folder layout adjustment though.

I didn't cover the part about pushing to repo up to GitHub, which seems to be your last 3 commands.

Offline leeor_net

  • Administrator
  • Hero Member
  • *****
  • Posts: 2352
  • OPHD Lead Developer
    • LairWorks Entertainment
Re: SVN -> Git Conversions
« Reply #3 on: September 15, 2017, 05:40:13 AM »
Right, URL_HERE is just the SVN repository's URL. Git clones the SVN with full history into a local directory. I then apply a filter based on file/directory (in some cases have to do it several times... there's probably a better way to do it but this works). I then compress it to filter out any revision history that is now empty, then attach the local repository to a remote repository and push/sync it with remote master.

Your command set looks to rework the directory structure. That could be done after the clone and either before or after filtering out files.

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
Re: SVN -> Git Conversions
« Reply #4 on: September 30, 2017, 12:50:10 AM »
Vagabond was asking about putting OP2MapImager up onto GitHub. I thought I'd document the process.

The OP2MapImager project did not have the standard SVN trunk/tags/branches folder structure, so no special conversion was needed there. OP2MapImager had the standard format OP2 tilesets as part of the project. Since these were proprietary, they needed to be filtered out. This removes the files not just from the HEAD revision, but also from all previous commits, ensuring there is no trace left in the repository. The project also previously had some maps for test data. The test data was removed by a previous commit, but the map files still existed in history. Again, since they were proprietary, they needed to be filtered out. The project also contained 7za.exe and the FreeImage library. Checking the licensing files, it appears they both allow for redistribution, and so they were left in with no changes.

The conversion steps were:
Code: [Select]
mkdir OP2MapImager
cd OP2MapImager/
git svn init https://svn.outpostuniverse.org:8443/svn/outpost2/GameResources/OP2MapImager/
git svn fetch
git filter-branch --tree-filter "rm -f OP2MapImager/*.bmp" --prune-empty HEAD
git filter-branch -f --tree-filter "rm -rf OP2MapImager/TestFiles/" --prune-empty HEAD

To upload the results to GitHub, a GitHub repository needs to be created first. This can be done from the GitHub website. To resulting URL was: https://github.com/OutpostUniverse/OP2MapImager.git

To upload the contents of the local repository to the new empty GitHub repository:
Code: [Select]
git remote add origin https://github.com/OutpostUniverse/OP2MapImager.git
git push -u origin master


The project is now up on GitHub. The URL for the page is:
https://github.com/OutpostUniverse/OP2MapImager

« Last Edit: September 30, 2017, 12:52:57 AM by Hooman »

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1015
Re: SVN -> Git Conversions
« Reply #5 on: September 30, 2017, 04:15:25 AM »
Hooman, thanks for uploading OP2Imager to Github. I just downloaded it, compiled it, and created the gitignore file.

We will have to decide how to deal with the tilesets missing from the source code on GitHub. Ideally it would be contained in the game download, but I'll try hosting it on the Wiki temporarily. The ReadMe can include where to fetch them. For the next release of Outpost 2, we should consider including the non Outpost 2 specific tileset images since the game runs fine with generic BMPs. This way the ReadMe can just focus on telling the user to get them from the Outpost 2 download. Probably worth discussing in more detail in a different thread if needed to keep from hijacking the purpose of this one.

I'll also look at collapsing the project and solution file into the same directory so the ReadMe file exists in the root directory instead of having to drill down a directory. This will make for a nicer experience browsing the code on GitHub.

-Brett

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
Re: SVN -> Git Conversions
« Reply #6 on: October 01, 2017, 02:22:51 AM »
We should also look into using Git submodules. That's something I avoided for now, just to get the project up there. Ideally, the OP2Utility project should be added as a submodule to both the OP2MapImager project, and the OP2Archive project. Since submodules are tied to a commit and only updated manually, that also means you can update the OP2Utility project without breaking the other projects. Right now, it's left to the developer to clone both projects individually, and set appropriate paths for building.

The tilesets should probably be zipped and posted to the forum somewhere. Good idea to repack them for the next release. Might also be incentive to get the native OP2 format bitmap loading code done, though I'm still in favour of repacking.

I agree with collapsing the project structure and putting the ReadMe in the root folder. Having the ReadMe.md file appear on the repo page does improve the GitHub experience.
« Last Edit: October 01, 2017, 03:06:24 AM by Hooman »

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
Re: SVN -> Git Conversions
« Reply #7 on: November 26, 2017, 02:58:07 PM »
I spent a bit more time this weekend looking into repository conversions.

GitHub has instructions on Splitting a Subfolder Out Into a New Repository. This could be used on a full SVN -> Git repository conversion to split it into individual projects. The main tool is the filter-branch command with the --subdirectory-filter option:
Code: [Select]
git filter-branch --prune-empty --subdirectory-filter FOLDER-NAME  BRANCH-NAME

Another point of interest, is the git-svn command can combine the git svn init and git svn fetch commands into a single git svn clone command.

I also tried experimenting with breaking the --stdlayout flag into the individual -b (--branches), -t (--tags), and -T (--trunk) flags:
Code: [Select]
git svn clone -b branches/API/Outpost2DLL/ -t tags/API/Outpost2DLL/ -T trunk/API/Outpost2DLL/ https://svn.outpostuniverse.org:8443/svn/outpost2/LevelsAndMods/ Outpost2DLL

There is a tagged release of the SDK that I wanted to preserve. There was also a branch I wanted to preserve, though realized much later the branch was only for the OP2Helper project, and not for Outpost2DLL. Because of this confusion, my expectations were off, and so I wasn't seeing the results I expected. This lead to a lot of extra reading, and experimenting with various flags, such as the ones shown above.

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
Re: SVN -> Git Conversions
« Reply #8 on: November 27, 2017, 03:04:57 AM »
Ok, I think I found a way of getting rid of a lot of the ugliness with a project that has been moved multiple times in the original SVN repo. The ugliness being quite a few revisions in the imported history with silly trivial changes, such as files being moved in and out of "trunk", "tag", and "branch" folders. This is an artifact of how SVN stores branches and tags, rather than real project changes. If you move a project folder within the repository layout, you're changing the repository layout, not the project. As such, this noise is not really desirable, as it doesn't reflect any changes to the project, and so should be filtered out along with other external changes, such as modifications to other projects.

It seems it's possible to filter out these changes by specifying multiple trunk, tag, and branch options when doing the conversion. You just have to know the history of where the project has existed in the source repo, and specify each location.

To get the history of project locations, I cheated a bit, and used a full SVN -> Git repo conversion that contains all projects. I then searched and filtered through all paths over all commits to get a quick list of where the project of interest was over time. For the OP2Helper project this could be one by searching for the main header file:

Code: [Select]
git log --pretty=format: --name-only --diff-filter=A | sort -u | grep OP2Helper.h

This produced the following list of files, from which paths could be extracted and classified:
Code: [Select]
API/HFL/SDKInclude/OP2Helper.h
API/OP2Helper/branches/BaseBuilderV2/OP2Helper.h
API/OP2Helper.h
API/OP2Helper/OP2Helper.h
API/OP2Helper (tag - SDK-V2)/OP2Helper.h
API/OP2Helper/tags/OP2Helper - SDK-V2/OP2Helper.h
API/OP2Helper/tags/OP2Helper (tag - SDK-V2)/OP2Helper.h
API/OP2Helper/trunk/OP2Helper.h
LevelsAndMods/API/OP2Helper/branches/BaseBuilderV2/OP2Helper.h
LevelsAndMods/API/OP2Helper/tags/OP2Helper - SDK-V2/OP2Helper.h
LevelsAndMods/API/OP2Helper/trunk/OP2Helper.h
LevelsAndMods/branches/BaseBuilderV2/OP2Helper/OP2Helper.h
LevelsAndMods/tags/SDK-V2/OP2Helper/OP2Helper.h
LevelsAndMods/trunk/API/OP2Helper/branches/BaseBuilderV2/OP2Helper.h
LevelsAndMods/trunk/API/OP2Helper/OP2Helper.h
LevelsAndMods/trunk/API/OP2Helper/tags/OP2Helper - SDK-V2/OP2Helper.h
LevelsAndMods/trunk/API/OP2Helper/trunk/OP2Helper.h
ZZJ-misc/Include/OP2Helper.h

Then to import the history:
Code: [Select]
git svn clone -b LevelsAndMods/trunk/API/OP2Helper/branches/ -b LevelsAndMods/branches/*/OP2Helper/ -b LevelsAndMods/API/OP2Helper/branches/ -b API/OP2Helper/branches/ -t LevelsAndMods/trunk/API/OP2Helper/tags -t LevelsAndMods/tags/*/OP2Helper/ -t LevelsAndMods/API/OP2Helper/tags/ -t API/OP2Helper/tags/ -T LevelsAndMods/trunk/API/OP2Helper/ -T LevelsAndMods/API/OP2Helper/trunk/ -T API/OP2Helper/trunk/ https://svn.outpostuniverse.org:8443/svn/outpost2/ OP2Helper

Checking the history of the resulting project repository:
Code: [Select]
git log --pretty=format: --name-only --diff-filter=A | sort -u

And the history comes out quite clean. No trunk, tag, or branch folders:
Code: [Select]
BaseBuilder.cpp
BaseBuilder.h
BaseBuilderV2.h
EnumSoundID.h
EnumTechID.h
Lib/OP2Helper.lib
OP2Helper.cpp
OP2Helper.dsp
OP2Helper.dsw
OP2Helper.h
« Last Edit: November 27, 2017, 03:07:21 AM by Hooman »

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
Re: SVN -> Git Conversions
« Reply #9 on: November 28, 2017, 03:41:43 PM »
I did a bit more tweaking to the OP2Helper project before uploading it.

I filtered out the OP2Helper.lib file, which should never have been added to source control in the first place. This project is a static library, so the lib file is the output, much like a DLL or EXE is the output of other projects.
Code: [Select]
git filter-branch --index-filter 'git rm --cached --ignore-unmatch -r Lib' --prune-empty --tag-name-filter cat -- --all

Note the Outpost2DLL project does contain a lib file, but that is used as an input to link to Outpost2.exe. The Outpost2DLL.lib file would essentially be an output when compiling Outpost2.exe, which we don't have source code for. This lib file had to be generated through other means, from the Outpost2 executable file itself. Hence why the Outpost2DLL contains a .lib file in the repository, but other projects don't.

I searched history to find possibly missed traces of .lib files, perhaps outside of the Lib folder. To get a fairly raw dump of what the repository is holding, I found rev-list to be useful:
Code: [Select]
git rev-list --objects --all

I also came across a way to get a list of the 10 largest files in the repository. A good way to know if somewhere in history you've accidentally committed something big, which is eating up a lot of repository space. First run git gc to generate pack files. Then combine rev-list with verify-pack. This outputs a list of hashes and file names:
Code: [Select]
git gc
git rev-list --objects --all | grep "$(git verify-pack -v .git/objects/pack/*.idx | sort -k 3 -n | tail -10 | awk '{print$1}')"

I found a command that lists hashes, file sizes, and file names, all sorted by size. It's a bit more complicated, but the output is very nice:
Code: [Select]
git rev-list --objects --all | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | awk '/^blob/ {print substr($0,6)}' | sort --numeric-sort --key=2 | cut --complement --characters=13-40 | numfmt --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest



I took a look at the tags. It seems two branches were created for the SDK-V2 tag. In Subversion, tags are actually full branches, and you can commit to them, though it warns you not to. Nevertheless, someone did indeed commit to the SDK-V2 branch, which is why the tag showed up twice during the conversion.

I tagged the original branch point as "v2.0.0". I tagged the most recent SVN version as "v2.1.0". For the commit that was erroneously added on top of the tag branch in SVN, I tracked back to it's proper branch point, created a new branch, cherry-picked the changes over to the new branch as the first commit. I then updated the code to make the style a bit more consistent with the rest of the SDK as a second commit. The changes were then merged it in to master branch as a merge commit.

During all this, I found my editor was trying to make changes to unexpected lines. Eventually I figured out it was cleaning up sloppy whitespace. Things like spaces and tabs at the end of lines. Rather than blend the changes in with the rest of the merge, or fight it indefinitely, I opted to cleanup the whitespace for the whole project in an additional commit.

I also investigated the BaseBuilderV2 branch. I found the code had all been committed, seemingly by accident, in various out of order stages at later dates, to trunk in SVN. Hence the branch contained no additional code. After verifying it really was all already on master, I chose to just discard the branch.



Well, that was a long convoluted learning process. The OP2Helper conversion is finally up on GitHub though.

Now that I know much more about what I'm doing, I think I might want to revisit the Outpost2DLL project. I encountered some of the same problems there, with the noisy history from the project being moved around during SVN repository layout updates. It might be worth redoing that conversion before people start using it.

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
Re: SVN -> Git Conversions
« Reply #10 on: November 30, 2017, 10:32:16 AM »
Ok, after more fighting with Outpost2DLL, the conversion process is still not working the way I'd like. Worse, after truncating history about midway, and not realizing it for some time, I thought I'd go back and check the OP2Helper conversion and realized I did it there too. Project history got cut about half way during a sloppy project folder move in SVN. It broke tracking when all the files were moved, rather than the containing folder. No changes beyond that point were imported to Git.

Before the mess gets any worse, I went and deleted the two repositories I'd uploaded to GitHub.

Thankfully I'm keeping detailed notes of all the things I've tried. Eventually I'll get there.  :'(

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
Re: SVN -> Git Conversions
« Reply #11 on: November 30, 2017, 06:10:56 PM »
Ok, Outpost2DLL has been converted, fixed, and pushed up to GitHub. Everything seems to be right this time. Check it out.



We should discuss tagging at some point. I tagged the initial commit as "v2.0.0", which corresponds with the SDK-V2 release. I was thinking of also tagging the last commit from SVN. Not sure how appropriate that is, and if it should be tagged as "v2.1.0" or "v3.0.0". Assuming we go with semantic version numbers anyway. Maybe we should stick to something simpler. Also, the SDK was previously packaged as a set of projects, with one version number for all of them. Now, the projects will exist independently in Git, so we might consider separate version numbers.

There's also the Git distinction between simple tags and annotated tags. That's an issue I'd like to delay a bit on for now. I used a simple tag, but perhaps I'll switch it to an annotated tag at some point. I saw one piece of advice that suggests simple tags are more appropriate for local use, while annotated tags are more appropriate for shares use in the master repository.



As for the conversion. As I added more and more options to git-svn clone, I found it did less and less of what I wanted it to do. Eventually I gave up on trying to get git-svn clone to do the full conversion in one step, and just used a simpler clone command to get a basic messy conversion done, and then cleaned it up a bit with a few filter commands to rewrite history and remove the odd Subversion artifacts.

The result contains the root of the project as the root of the repository, with no spurious trunk or tag folders floating around in history. The history contains all changes to the project, and I triple checked SVN revision numbers on this one. Every commit contains changes to project code. There are no empty commits. There are also no commits with changes external to the project, such as the project as a whole being moved around within the SVN repository, nor other projects being copied into the project folder or otherwise having files side-by-side in the same folder. There were points in history where the same folder was used by multiple projects.

One difference between Git and SVN, is Git only tracks files, not folders. Existence of folders is implied by paths of stored files. If an SVN commit only adds an empty folder, that commit will appear empty after being imported into Git. Similarly, if you move all the files out of a folder, the folder will disappear in Git. There was a bit of that going on throughout history, such as adding a placeholder folder in one commit, then moving files in a second commit. In particular, the first commit did this, though wasn't automatically pruned by the other commands.



The conversion worked something like this:
Code: [Select]
# Initial SVN -> Git import
git svn clone \
-T LevelsAndMods/trunk/API/Outpost2DLL/ \
-t LevelsAndMods/trunk/API/Outpost2DLL/tags/ \
https://svn.outpostuniverse.org:8443/svn/outpost2/ Outpost2DLL
cd Outpost2DLL

# Remove empty commits
git filter-branch --prune-empty master
# Remove any "tags" folders
git filter-branch --index-filter 'git rm -rf --cached --ignore-unmatch tags/' --prune-empty -f -- --all
# Move any files in "trunk" folders back to root
git filter-branch --tree-filter 'test -d trunk/ && mv trunk/* ./ || echo "Nothing to do"' --prune-empty -f -- --all
# Remove the empty initial commit
git rebase -i --root

# Tag the initial commit, which was the SDK-V2 release
git tag v2.0.0 `git rev-list --max-parents=0 HEAD`

# Push changes up to GitHub
git remote add origin https://github.com/OutpostUniverse/Outpost2DLL.git
git push origin v2.0.0

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1015
Re: SVN -> Git Conversions
« Reply #12 on: November 30, 2017, 07:15:01 PM »
Hooman,

Thanks for putting all the time into gettings OP2Helper and Outpost2DLL into the repository.

It will be nice to have them available with all the history intact.

I appreciate removing the SVN specific each tag goes in it's own directory when tranferring over.

In the long term, is there anything stopping HFL from making its way onto Git? I was excited to realize that my scenarios could be transferred over, but at least some if not all depend on HFL as well.

-Brett

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
Re: SVN -> Git Conversions
« Reply #13 on: December 03, 2017, 10:50:02 PM »
Ok, the fixed OP2Helper conversion has been re-uploaded to GitHub. Check it out.

A Pull Request was created for some code that had been erroneously committed to a tag (which in SVN are actual branches). It's use is a bit dated, but still good. It can be auto merged in. I figure I'll leave someone else to review it and make a decision to include in master or not.



The conversion:
Code: [Select]
# Initial conversion from SVN to Git
git svn clone \
  -T LevelsAndMods/trunk/API/OP2Helper/ \
  -t LevelsAndMods/tags/*/OP2Helper/ \
  -b LevelsAndMods/branches/*/OP2Helper \
  https://svn.outpostuniverse.org:8443/svn/outpost2/ OP2Helper
cd OP2Helper

# Remove the initial blank commit
git rebase -i --root
# Move spurious copies in trunk back to project root
git filter-branch --tree-filter 'test -d trunk/ && mv trunk/* ./ || echo "Nothing to do"' --prune-empty -f -- --all
# Remove spurious tag subfolders
git filter-branch --index-filter 'git rm --cached --ignore-unmatch -r tags' --prune-empty --tag-name-filter cat -f -- --all
# Remove spurious branches subfolders
git filter-branch --index-filter 'git rm --cached --ignore-unmatch -r branches' --prune-empty --tag-name-filter cat -f -- --all
# Remove erroneously committed copies of the compiled lib file
git filter-branch --index-filter 'git rm --cached --ignore-unmatch -r Lib' --prune-empty --tag-name-filter cat -f -- --all

# Tag the initial commit
git tag v2.0.0 `git rev-list --max-parents=0 HEAD`

A few updates were made to master to fix minor issues. Branch updates for a lost commit to a tag were brought over from a previous conversion attempt.
Code: [Select]
# Branch for ZZJ's commit, based on tagged release
git checkout -b centerView v2.0.0
# Bring in changes from previous conversion attempt
git fetch ../OP2Helper-old/ master
# Cherry pick changes (ZZJ's commit, and updates to code style)
git cherry-pick FETCH_HEAD~3^2^
git cherry-pick FETCH_HEAD~3^2
# Handle merge conflict
git merge master

After that, the results were uploaded to GitHub, and a Pull Request was created.
Code: [Select]
git remote add origin https://github.com/OutpostUniverse/OP2Helper.git
git push -u origin master
git push -u origin centerView
git push origin v2.0.0



HFL seems like a sensible next project to convert.

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
Re: SVN -> Git Conversions
« Reply #14 on: December 06, 2017, 10:10:30 PM »
HFL and HFL-IUnit projects have been converted and uploaded to GitHub.

Conversion steps:
Code: [Select]
git svn clone -T LevelsAndMods/trunk/API/HFL/ https://svn.outpostuniverse.org:8443/svn/outpost2/ HFL
git svn clone -T LevelsAndMods/trunk/API/HFL-IUnit/ https://svn.outpostuniverse.org:8443/svn/outpost2/ HFL-IUnit

Both projects had history filtered to remove empty commits, and .lib files.

For HFL:
Code: [Select]
git rebase -i --root
git filter-branch --index-filter 'git rm --cached --ignore-unmatch -r Lib' --prune-empty --tag-name-filter cat -f -- --all

For HFL-IUnit:
Code: [Select]
git rebase -i --root
git filter-branch --index-filter 'git rm --cached --ignore-unmatch -r op2extra.lib' --prune-empty --tag-name-filter cat -f -- --all



After that, projects were uploaded to GitHub.
For HFL:
Code: [Select]
git remote add origin https://github.com/OutpostUniverse/HFL.git
git push -u origin master

For HFL-IUnit:
Code: [Select]
git remote add origin https://github.com/OutpostUniverse/HFL-IUnit.git
git push -u origin master
« Last Edit: December 06, 2017, 10:14:14 PM by Hooman »

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1015
Re: SVN -> Git Conversions
« Reply #15 on: December 07, 2017, 02:13:34 AM »
Hooman,

Thank you for uploading HFL.

I think the last prerequsitie my scenarios take is Outpost2Dialog. This project is basically odasl and a sample of how to create a dialog box at the beginning of a scenario. It also shows how to set/fetch resources to feed the dialog box a message.

I'm not sure how I feel about this project. odasl is used in a lot of scenarios, so it shouldn't be included separately in them all. But, maybe it makes more sense to place it in Outpost2DLL or somewhere else as opposed to being a separate project. I don't know?

The number of projects included on GitHub is starting to grow quite a bit. :)

-Brett

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
Re: SVN -> Git Conversions
« Reply #16 on: December 07, 2017, 04:19:52 AM »
Quote
The number of projects included on GitHub is starting to grow quite a bit. :)

Yes. I think BlackBox once suggested the Outpost2DLL, Outpost2App, and possibly even OP2Helper projects should be merged. I think I'm starting to see the light on that one. I've been considering lately they can all be separate folders within one project, with one master header file that includes sub-master header files for each section.


Where is Outpost2Dialog?

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1015
Re: SVN -> Git Conversions
« Reply #17 on: December 07, 2017, 01:46:08 PM »
https://svn.outpostuniverse.org:8443/!/#outpost2/view/head/LevelsAndMods/trunk/API/Outpost2Dialog

I'm not sure either way about combining the projects. op2ext depends on Outpost2DLL but doesn't need any code from OP2Helper. Reducing the number of projects could be a good thing though.

-Brett

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
Re: SVN -> Git Conversions
« Reply #18 on: December 12, 2017, 10:32:17 PM »
Ahh, thanks. Seems I had a git-svn branch checked out from a point before that project was added, and didn't see it. (The branch was the point before the modification to Outpost2.exe that messed up the OllyDbg comment tracking).

I see the files now with:
Code: [Select]
svn log https://svn.outpostuniverse.org:8443/svn/outpost2/LevelsAndMods/trunk/API/Outpost2Dialog -v



Quote
I'm not sure either way about combining the projects. op2ext depends on Outpost2DLL but doesn't need any code from OP2Helper. Reducing the number of projects could be a good thing though.

I think that was my original reasoning for the current project organization. I really wanted to keep things separate. Don't add dependencies you're not using, and don't mix our additions with original Outpost 2 interface stuff. Though what I was thinking now, is separate folders within one project. Perhaps something like:
Outpost2DLL\
  game\  (Formerly Outpost2DLL)
  shell\  (Formerly Outpost2App)
  opu\  (Formerly OP2Helper)

In terms of compiled size, the linker will strip out anything that's not used, so combining projects should not affect final output.

There are some usability concerns this would address. The project settings adjustments needed to add a new API project can be confusing for new people. After all, there is a wiki article on this topic. ;) By combining it into one standard project, which a standard template comes preconfigured to use, they have exactly what they need, without worrying about configuration, and without having to select the right template to get all the dependent projects they need.


Such a change means we'd have a proper static library project, rather that a pseudo header-file-only static library project. Either which way, you still need to link a .lib file, since these all depend on the .lib for Outpost2.exe. It was never a true header-only project to begin with.
« Last Edit: January 05, 2018, 06:13:27 AM by Hooman »

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
Re: SVN -> Git Conversions
« Reply #19 on: January 07, 2018, 12:32:20 PM »
I converted the projects in LevelTemplates from the SVN repo and put them up on GitHub. These projects contain relative references to the API projects. We should convert them to use Git submodules. Once that is done, we can look into updating instructions to use the GitHub template projects to build new levels.



Projects converted:
LevelTemplate-Blank
LevelTemplate-Blank-WithComments
LevelTemplate-Hooville

During the conversion, I experimented with the Hub command line tool to create projects on GitHub without using the GitHub web interface. It seems Hub still defaults to use SSH rather than HTTPS. There are plans to change the default to HTTPS. Meanwhile, to keep things consistent, Hub can be configured to use HTTPS with:
Code: [Select]
git config --global hub.protocol https

An example conversion looks something like this:
Code: [Select]
git svn clone https://svn.outpostuniverse.org:8443/svn/outpost2/ -T LevelsAndMods/trunk/LevelTemplates/Hooville Hooville
cd Hooville
git rebase -i --root
git rev-list --objects --all | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | awk '/^blob/ {print substr($0,6)}' | sort --numeric-sort --key=2 | cut --complement --characters=13-40 | numfmt --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest
git filter-branch --index-filter 'git rm --cached --ignore-unmatch -r trunk/' --prune-empty --tag-name-filter cat -- --all
git log --oneline --name-status
git filter-branch --index-filter 'git rm --cached --ignore-unmatch -r tags/' --prune-empty --tag-name-filter cat -f -- --all

# Create project on GitHub
hub create -d "Hooville multiplayer template mission" OutpostUniverse/LevelTemplate-Hooville
# This should push up to GitHub, though a failed previous attempt may have set SSH protocol
#git push origin master

# To fix the SSH -> HTTPS settings I used
git remote remove origin
git remote add origin https://github.com/OutpostUniverse/LevelTemplate-Hooville.git
git push -u origin master

Offline Vagabond

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1015
Re: SVN -> Git Conversions
« Reply #20 on: January 09, 2018, 12:19:35 AM »
That is a pretty lengthy command line switch. Nice job.

I've been thinking some more about how to organize large projects/solutions/workspaces on GitHub. I've noticed that it seems common to include extras along with the main project, like templates, examples, test code, plugins, etc.

Perhaps it would be better to bundle Outpost2DLL, OP2Helper, I-Unit, and HFL, and the 3 test projects in one repository.

Something like the folders:
Outpost2DLL
OP2Helper
Templates
 -subfolders in Templates X, Y, Z

Then the root directory would just contain the readme, .gitignore, license, etc.

No one is going to want OP2Helper or the templates without Outpost2DLL. The only downside is, now for a project like op2ext, you have to download extra / unnecessary files. However, you would still only be compiling against the Outpost2DLL project.



Perhaps more concrete, but I think the NetFix client and server should be bundled as a single repository. I'm unlikely ever going to download the source for either without wanting to reference the other.

Anyways, I'm glad to see all the history being cared for so nicely. Especially considering this use of Git is beyond my league.

-Brett

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
Re: SVN -> Git Conversions
« Reply #21 on: January 10, 2018, 05:05:27 AM »
Heh, the really long line is copy/pasted from the web, I take no credit for it. I introduced it earlier in the thread when I first found it. Very handy for getting a list of paths throughout the history of the repository, so you know if there are any "trunk/" or "tag/" paths in the history. I originally used it to find large files, such as accidentally committed .lib files.


You make some good points concerning project and repository organization. I would like to combine some of them. I've been a bit conservative about doing so in the past. Certainly the NetFix client and server make sense to combine.

I'm still hesitant to combine the templates with the API. There is one simple reason for this. I see people starting new projects by forking the templates. When they are bundled together, it's far too easy and tempting to make quick fixes to the API, and then not contribute them back to the original source. People probably won't be clear on how to contribute API changes back to the original after forking a repository and beginning work on their own project. The API changes then get lost within the individual projects. If the API was a submodule though, there is a clear boundary, and it's very clear where each set of changes gets committed to. It also discourages inappropriate changes, such as project specific hacks made to the API, as such changes wouldn't be accepted upstream.

Another concern I have, and this one gets a bit conservative, is some projects make a lot of assumptions about the internal structure of Outpost2.exe. Levels built according to these APIs are not guaranteed to work with different versions of Outpost 2. If someone wants to use a CD install, or a different language version of Outpost 2, those levels quite possibly won't work, and may likely crash or corrupt their game. The same might be true if we make sufficient modifications to the executable file. This leave me thinking we should divide API projects up into two categories, one basic set that only interfaces to the game using it's official exported API (Outpost2DLL, Outpost2App, OP2Helper), and an advanced set that uses memory hacking and code patching (HFL, IUnit, ForcedExports).

I was also thinking ForcedExports needs to be renamed, possibly to something like OP2Internal. I'll probably rename it when I convert it.

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
Re: SVN -> Git Conversions
« Reply #22 on: June 27, 2018, 02:11:38 AM »
I went back to revisit the NetFixServer and NetFixClient projects. They had been imported early and still had the messy trunk and tag folders. They also hadn't yet been touched or added to on GitHub, so I went ahead and redid the imports for those projects to clean them up a bit.



For NetFixServer, I'd already tested and did a proper import ages ago, but had never got around to uploading and overwriting the existing repo. I think at the time I was still a little uncertain of what I was doing with Git and GitHub. To upload the more recently converted repo, the steps were:
Code: [Select]
git remote add origin https://github.com/OutpostUniverse/NetFixServer.git
git push -u origin master -f
git push origin v1.0.0

I already had a checked out version of the old repo, which other people might too. To update things on the client side after a forced push, I used:
Code: [Select]
git fetch
git reset origin/master
git checkout .
git clean -f

The clean didn't seem to remove old folders, so I manually deleted those.



For the NetFixClient import, the steps were:
Code: [Select]
# Initial import
mkdir NetFixClient
cd NetFixClient
git svn init https://svn.outpostuniverse.org:8443/svn/outpost2/NetFix/Client/ --stdlayout
git svn fetch

# Checked repo for stray "trunk" or "tag" folders (courtesy of StackOverflow)
git rev-list --objects --all | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | awk '/^blob/ {print substr($0,6)}' | sort --numeric-sort --key=2 | cut --complement --characters=13-40 | numfmt --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest

# Checked for empty commits to filter out (there were none)
git rev-list --count HEAD     # 26
git rebase -i --root
git rev-list --count HEAD     # 26
git filter-branch --index-filter 'git rm --cached --ignore-unmatch -r trunk/' --prune-empty --tag-name-filter cat -- --all
git rev-list --count HEAD     # 26

# Clean up an old empty commit message
git rebase -i --root
# => Used "reword" on the offending message, and added a description of the change

# View branches to see SVN tags
git branch -a -v

# Convert SVN branches into Git tags
# Note: The commit message change above rewrote history.
# This step required matching up previous commits with new commit IDs by examining log messages
git tag -a 2011-11-18 15c6001
git tag -a 2008-09-27 a7e00a6

# Pushing changes up to GitHub
git remote add origin https://github.com/OutpostUniverse/NetFixClient.git
git push -u origin master -f
git push origin 2008-09-27 2011-11-18


Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
Re: SVN -> Git Conversions
« Reply #23 on: January 23, 2019, 06:54:25 AM »
I looked into the steps needed to convert the ForcedExports project from SVN to Git. During the conversion, I'm planning to rename it to OP2Internal.

As the SVN server is currently down, I've been testing the steps using a local backup. This means references and comments in my experiment point to a local copy, rather than the official one. I'm planning to either re-run the steps from the official repository, or perhaps find a way to rename the references.

Conversion steps (though my initial clone command used a modified source):
Code: [Select]
# Initial clone (and folder rename)
git svn clone https://svn.outpostuniverse.org:8443/svn/outpost2/ -T LevelsAndMods/trunk/API/ForcedExports OP2Internal

# Move spurious copies in trunk back to project root
git filter-branch --tree-filter 'test -d trunk/ && mv trunk/* ./ || echo "Nothing to do"' --prune-empty -f -- --all
# Remove accidentally committed .obj file
git filter-branch --index-filter 'git rm --cached --ignore-unmatch -r Build/' --prune-empty --tag-name-filter cat -f --

# Reword commit messages...
git rebase -i --root

Side note: I'm really glad I created this thread, as this stuff gets really hard to remember after a while.

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4955
Re: SVN -> Git Conversions
« Reply #24 on: January 27, 2019, 05:43:44 PM »
The old "ForcedExports" project has been uploaded to GitHub as "OP2Internal".

After re-running the above steps from the now up and running SVN repository, the final upload commands were:
Code: [Select]
git remote add origin https://github.com/OutpostUniverse/OP2Internal.git
git push -u origin master

Hopefully this unblocks some needed NetFix development.