Author Topic: OP2MapImager Development  (Read 732 times)

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 3810
Re: OP2MapImager Development
« Reply #25 on: July 02, 2017, 06:35:44 AM »
Nice job with the documentation.

We'll have to get the custom OP2-format BMP loading code working, as well as loading directly from VOL files. I consider the direct VOL loading to be higher priority, since you'd need to read the VOL files before getting access to the custom format BMP files stored inside. Of course that could just be me procrastinating on touching the messy custom format BMP loading code.

The release build steps seem long and manual. Perhaps we should script that. Maybe we could create a new project in the solution space for building the release package. It could depend on the project, so building it would automatically rebuild the project, and fail if the project failed to build. Maybe throw the commands into the post build step to copy files to the right folder and zip them up. I don't know how you want to handle version numbers, though it would be easy enough to append a date to the archive name.

Offline Vagabond

  • Sr. Member
  • ****
  • Posts: 356
Re: OP2MapImager Development
« Reply #26 on: July 03, 2017, 03:50:22 PM »
Hooman,

I spent some time learning more about post build events in Visual Studio. Below is the result. Now Visual Studio will check if the code was built in release mode, and if so create a folder called 'OP2MapImager X.X' inside the release folder. Then it populates the new folder with all the files required for the build.

The user will still have to manually change the version number in the code, folder name, and ReadMe before zipping the folder. So, I guess it eliminates some of manual labor.

Thank you for the reply in the other topic on expanding VolDecompress. I haven't had time to implement it yet, but should be next on the list to look at. I would prefer moving away from the WIN API code, but I'm not sure it makes sense for me to just rewrite a smattering of the functions this way and leave the rest with WIN API and COM interoperability.

x86 vs x64 compatibility
I think the WIN API code used in OP2VolDecompress is not compatible with x64 builds. I can only seem to get it to compile in x86.

Code: [Select]
xcopy /y /d "$(ProjectDir)FreeImage\FreeImage.dll" "$(OutDir)"

if $(ConfigurationName) == Release (
    mkdir "$(OutDir)OP2MapImager X.X"
    xcopy /y /d "$(ProjectDir)well00*.bmp" "$(OutDir)OP2MapImager X.X"
    xcopy /y /d "$(ProjectDir)FreeImage license-gplv3.txt" "$(OutDir)OP2MapImager X.X"
    xcopy /y /d "$(ProjectDir)ReadMe.txt" "$(OutDir)OP2MapImager X.X"
    xcopy /y /d "$(OutDir)FreeImage.dll" "$(OutDir)OP2MapImager X.X"
    xcopy /y /d "$(OutDir)$(TargetFileName)" "$(OutDir)OP2MapImager X.X"
)

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 3810
Re: OP2MapImager Development
« Reply #27 on: July 05, 2017, 06:39:00 AM »
For the X.X, perhaps your script can set a shell variable using the "svn info" command. The command on it's own will list a number of values, but you can select one specifically with the --show-item argument. I would recommend experimenting with the "last-changed-revision" value. You could tag each release with the SVN revision number. I believe by setting the path/url to the root of your project, it will include the most recent revision change to that project (ignoring more recent repository changes outside of your project). Of course if you depend on another project, it might be easier to just use "revision" (the global value). Another possibly useful value is "last-changed-date".

Code: [Select]
info: Display information about a local or remote item.
usage: info [TARGET[@REV]...]

  Print information about each TARGET (default: '.').
  TARGET may be either a working-copy path or URL.  If specified, REV
  determines in which revision the target is first looked up.

  With --show-item, print only the value of one item of information
  about TARGET. One of the following items can be selected:
     kind                  the kind of TARGET
     url                   the URL of TARGET in the repository
     relative-url          the repository-relative URL
     repos-root-url        the repository root URL
     repos-uuid            the repository UUID
     revision              the revision of TARGET (defaults to BASE
                           for working copy paths and HEAD for URLs)
     last-changed-revision the most recent revision in which TARGET
                           was changed
     last-changed-date     the date of the last-changed revision
     last-changed-author   the author of the last-changed revision
     wc-root               the root of TARGET's working copy

...

Once you have some kind of version tag, I would also recommend zipping the package as part of the build step.


If the release package preparation is slow, it may be nice to have it separated out. Looks like you're only doing it for release mode, which should be good enough. If later you decide to add testing, which might also be done in release mode, the packaging could be move to a later step. That's why I suggested a separate release package project that depends on the actual project. The actual project would still be the default project for builds, so the release package build would only run when requested, and project dependency rules would ensure the actual build was also up-to-date when it runs. Not sure if this makes sense to do; just wanted to be clear on the idea.


The Win API code does have differences between 32-bit and 64-bit. Mostly though, it uses auto-expanding type names, so they're the appropriate type on each platform. Depending on how things are coded, it can often be source compatible between the two, though not likely to be binary compatible.

You shouldn't need to use the Win API. You can depend on the standard C++ library. It's mostly just file access code that depends on the Win API, which there is reasonably straightforward replacements to in the standard C++ library.

I didn't see an easy way to do binary memory stream in the standard C++ library. It may make sense to still use your own StreamReader classes. A custom FileStreamReader class can just delegate to standard C++ library file access classes. A custom MemorySreamReader class can just be built in raw C++ without any standard library or operating specific code.

The existing code for OP2Editor used the Win API for the FileStreamReader. The MemoryStreamReader was simple C++ code without use of libraries or the Win API. The code was quite simple in both cases.

Offline Vagabond

  • Sr. Member
  • ****
  • Posts: 356
Re: OP2MapImager Development
« Reply #28 on: July 08, 2017, 05:29:19 AM »
Another Update,

I spent some time learning more about the command prompt and writing batch programs, which is essentially what Visual Studio C++ Post Build Events are.

There is not a built in zip console command in Windows (strange I thought), so I settled on 7-Zip's console application. 7Zip is a fairly well known, free, and easy to use alternative. I placed 7za.exe in the repository, which means that others will be able to run the post build event without having to fetch the application manually (if they are running windows).

I had to download the console application portion of TortoiseSVN. Unfortunately, it does not default to installing the console application when you install TortoiseSVN. So, if others pull the source code, the post build script will blow up until they do the same if not already installed. Perhaps there is a way to place the executable in folder like I did with 7-zip?

The code below is the full post build script. Basically it

  • saves the filename and directory to a local variable
  • creates a directory for all release files
  • copies all required files to the release directory
  • zips the release directory using 7-zip (I'm using the standard zip format, not 7-zip's arguably better format)
  • deletes the initially created directory

Code: [Select]
xcopy /y /d "$(ProjectDir)FreeImage\FreeImage.dll" "$(OutDir)"

REM SET zipName="OP2MapImager Ver1.0" svn info --show-item revision --no-newline
SET zipName="OP2MapImager Ver1.0"
SET directoryName=$(OutDir)%25zipName%25

if $(ConfigurationName) == Release (
    mkdir %25directoryName%25
    xcopy /y /d "$(ProjectDir)well00*.bmp" %25directoryName%25
    xcopy /y /d "$(ProjectDir)FreeImage license-gplv3.txt" %25directoryName%25
    xcopy /y /d "$(ProjectDir)ReadMe.txt" %25directoryName%25
    xcopy /y /d "$(OutDir)FreeImage.dll" %25directoryName%25
    xcopy /y /d "$(OutDir)$(TargetFileName)" %25directoryName%25

    7za a -tzip $(OutDir)%25zipName%25.zip %25directoryName%25
    REM DEL /q %25directoryName%25\*.*
    RMDIR %25directoryName%25 /s /q
)

The only piece I need help on is setting the revision number from SVN. I cannot figure out how to set the contents of a local variable as a concatenation of a string and the output of the SVN console command.

Successfully working SVN console command:
Code: [Select]
svn info --show-item revision --no-newline

I want it to read something like:
Code: [Select]
SET zipName="OP2MapImager Ver1.0" svn info --show-item revision --no-newline


@Hooman,

Thanks for the reply in the other thread on vol decompression. Now that I'm done fiddling with the post build event (besides the issue noted), I'll head back over and try to implement the tips you gave.
« Last Edit: July 08, 2017, 05:31:52 AM by Vagabond »

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 3810
Re: OP2MapImager Development
« Reply #29 on: July 09, 2017, 12:00:06 PM »
This may be relevant:
https://stackoverflow.com/questions/17546016/how-can-you-zip-or-unzip-from-the-command-prompt-using-only-windows-built-in-ca
It seems there is a way to compress files to ZIP using Power Shell with newer versions of Windows. Power Shell can be called from a regular shell or batch file. I noticed an alternate methods towards the end that may work for older versions of Windows.

For the variable, this might help:
https://stackoverflow.com/questions/2323292/windows-batch-assign-output-of-a-program-to-a-variable
Looks a bit sick, but seems to work.


On Linux, you could just use either of the following:
Code: [Select]
export variableName="Some fixed string: `command args`"

Code: [Select]
export variableName="Some fixed string: $(command args)"

Offline Vagabond

  • Sr. Member
  • ****
  • Posts: 356
Re: OP2MapImager Development
« Reply #30 on: July 11, 2017, 02:38:49 AM »
While testing ways to get the directory name to display correctly with the SVN version number, I managed to set the directory to the current directory and deleted the majority of the OP2MapImager code.

Fortunately, it is all backed up to the repository and was quick to restore. (It may have all been in the recycle bin on the CPU as well, but didn't really check).

After that, I bought a book on the Windows Console and batch scripting. Currently working through the book's content. I figured it was time to actually learn what I am doing instead of hacking it constantly for post build events.

Unfortunately, the project is languishing a little as I could have just spent 2 minutes copying files over for a release instead of all these hours learning a new skill. It is something I've wanted to learn for a while though.

I don't think the subject is too complicated, so hopefully in a couple of days I'll have moved through the book far enough to know how loops, escape characters, etc. actually work and do things right.

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 3810
Re: OP2MapImager Development
« Reply #31 on: July 11, 2017, 09:55:40 PM »
Quote
While testing ways to get the directory name to display correctly with the SVN version number, I managed to set the directory to the current directory and deleted the majority of the OP2MapImager code.
Hah! :P

Quote
Unfortunately, the project is languishing a little as I could have just spent 2 minutes copying files over for a release instead of all these hours learning a new skill. It is something I've wanted to learn for a while though.
Welcome to the world of software development. Where people spends hours or days, to save a few minutes or seconds.  ::)

Funny how that works out too. You notice some boring repetitive task that keeps eating away at a few minutes, plus the associated cognitive load. So you figure you'll spend an extra two minutes to automate the two minute task. And it works, mostly, and saves time, but there is one small little largely inconsequential detail, and that's suddenly where you sink all your time and effort to get that last little bit done that keeps eating away at you. And you wonder out loud if it was worth the time, as you've now sunk more time into automating the task than the original task ever would have cost you in the long run. Yet you still feel victorious when you finally accomplish the task, because the hidden cost of that cognitive load is no longer weighing on you. And that makes it all worth it!  :)

Offline Vagabond

  • Sr. Member
  • ****
  • Posts: 356
Re: OP2MapImager Development
« Reply #32 on: July 15, 2017, 05:12:35 AM »
I finished the first of 3 books on the Windows command line/batch scripting. I figured out enough to pull in the SVN revision number and append it to the end of the directory name. The author separated his book into 3 volumes about 85 pages each. They sell for $3.00 each on Amazon for an electronic copy. I would have been willing to pay $9.00 for the book all together. https://www.amazon.com/Learn-Command-Line-Batch-Script-ebook/dp/B00V9D3QVM/

New post build script is below. New release workflow at the bottom of the ReadMe is posted below that. 

Code: [Select]
REM Place FreeImage DLL into debug or release directory for running application within Visual Studio.
xcopy /y /d "$(ProjectDir)FreeImage\FreeImage.dll" "$(OutDir)"

REM Pull repository revision number and set into final release directory name.
FOR /f %25%25i IN ('svn info --show-item revision --no-newline') DO SET svnVersion=%25%25i

SET zipNameBegining="OP2MapImager Ver1.0."
SET zipName=%25zipNameBegining%25%25svnVersion%25
SET directoryName=$(OutDir)%25zipName%25

if $(ConfigurationName) == Release (
    mkdir %25directoryName%25
    xcopy /y /d "$(ProjectDir)well00*.bmp" %25directoryName%25
    xcopy /y /d "$(ProjectDir)FreeImage license-gplv3.txt" %25directoryName%25
    xcopy /y /d "$(ProjectDir)ReadMe.txt" %25directoryName%25
    xcopy /y /d "$(OutDir)FreeImage.dll" %25directoryName%25
    xcopy /y /d "$(OutDir)$(TargetFileName)" %25directoryName%25

    REM Use 7Zip to zip final version
    7za a -tzip $(OutDir)%25zipName%25.zip %25directoryName%25
   
    DEL /q %25directoryName%25\*.*
    RMDIR %25directoryName%25 /s /q
)

Code: [Select]
 1. If changing Major/Minor revision number, set new version number at top of OP2MapImager main.cpp AND in post build event batch script.
 2. Run SVN Commit and then SVN Update to merge committed changes and properly update revision number of repository.
 3. Set Solution Configuration to Release.
 4. Set Solution Platform to x86. (x64 is not currently supported due to some WINAPI code within OP2Utility VolDecompress).
 5. Compile Code.
 6. The following files will automatically be copied into the zipped directory'OP2MapImager 1.0.XXXX' (XXXX is svn revision number):
    * OP2MapImager.exe (From Release Directory)
    * FreeImage.dll (x32 version)
    * Well0000.BMP-Well0012.BMP (Reformated BMPs that a normal image editor may open.)
    * ReadMe.txt (this file)
    * FreeImage liscense-gplv3.txt
 7. Place zip file on the Outpost Universe Website.

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 3810
Re: OP2MapImager Development
« Reply #33 on: July 15, 2017, 09:21:15 PM »
Nicely done. You're really going all in on this.

One small point, the "x.y.z" version numbers are generally referred to as Semantic Versioning. They have implied meanings concerning compatibility between versions. The revision numbers returned by SVN are more just a serial number. It might be better not to mix the two formats. Maybe just tag it as "rev-#". If you want something more human friendly, consider appending the revision date. You could append both a revision number and a revision date.

To get proper semantic version numbers, they would probably need to be manually tagged. One way is to detect a build from a tag branch, and extract the tag name. That only works for tagged releases though. I'd say just stick with revision numbers or dates for now.


I like that you actually bought a book to learn that stuff.
+1

Offline leeor_net

  • Administrator
  • Hero Member
  • *****
  • Posts: 1451
    • LairWorks Entertainment
Re: OP2MapImager Development
« Reply #34 on: July 18, 2017, 09:11:10 AM »

Implementation Details
 * Repository Type: Subversion
 * Language: C++
 * Windows IDE: Visual Studio 2017
 * Image manipulation library???

DevIL

Have been using it with great success in Rogue Arena.

Offline Vagabond

  • Sr. Member
  • ****
  • Posts: 356
Re: OP2MapImager Development
« Reply #35 on: July 18, 2017, 07:07:01 PM »
leeor_net,

We have had pretty good success using FreeImage. At this point, all the image manipulation is working properly, so I don't see any reason to switch, although I appreciate the suggestion. I hadn't updated the initial page to indicate we had went with it.

Hooman,

I read through the article on Semantic Versioning. It makes sense to me. I'll revisit this once I get the archive code to compile again (I'll post the errors in the VolDecompress thread).

I was thinking if there was a way to load the contents of a .txt file into a compile time constant, you could specify the version number in a text file. Then you could have the C++ program load the text file for indicating it's version number. It would be trivial to load the text file into the post build event and indicate it in the zipped filename and directory name. A build number within the version number could even be incremented within the text file via the post build event. Then whenever you were ready to change the version number, you would just need to change the .txt file and it would reflect in both the project and the post build event.

-Brett

Offline leeor_net

  • Administrator
  • Hero Member
  • *****
  • Posts: 1451
    • LairWorks Entertainment
Re: OP2MapImager Development
« Reply #36 on: July 19, 2017, 02:21:19 AM »
leeor_net,

We have had pretty good success using FreeImage. At this point, all the image manipulation is working properly, so I don't see any reason to switch, although I appreciate the suggestion. I hadn't updated the initial page to indicate we had went with it.

As always I'm late to the party. FreeImage is a good option too, I just happen to have inherited it with the Rogue Arena project and learned that it's quite good at what it does.