Remote Pair Programming

[Hooman]

Yes, TeamViewer worked pretty good. Lag was minimal, mostly only noticeable when I was trying to scroll to a specific point in a file, and even then not all that much.

TeamViewer can be used with or without registering an account with TeamViewer. It's easiest if you register an account with TeamViewer using your email address, and then add the other person to your friend's list. That allows you to easily send a request to join them using their computer, which they can accept or deny. There is another method we tried first, which didn't require registering an account with TeamViewer, and involved sharing access codes, similar to a username and password. Beware though that these access codes let someone join without prompting to accept on the host end. This method seems to be useful for accessing your own computer remotely, so you can join while it's unattended. To aid this feature, TeamViewer is also set to launch at startup by default when you install it. If you're security conscious, you might want to change that option, and perhaps not use the access code sharing method.

There was some trouble initially getting TeamViewer to use the mic on my end. It was only transmitting audio one way at first. We tried using Discord for the audio, but since TeamViewer was still transmitting audio one direction, including what Discord was playing, it caused my voice to echo back 1 or 2 words later. Very hard to speak clearly when you hear a delayed echo of yourself. It messes with your brain, confuses you, and makes you slur your words trying to compensate. Try it sometime, it's weird and fascinating. I eventually got the mic to work, though in hindsight I wonder if it may have had something to do with closing Skype first.

Chrome Remote Desktop did not connect for us. It failed during the connection attempt, after alerting the other side.

----

Stated project goals seem quite good. I've wanted something that can make scaled images of maps for a long time.

More generally, I would like new code to allow for easy cross platform extraction of game resources. Many of the current tools are Windows specific, and have extra dependencies that need to be installed. I'd like standalone utilities available with minimal dependencies, to ensure stuff just works wherever. Ultimately, I want to make it easy for anyone to access game resources, either through library code, or without needing to program anything, in easy to use standard formats. That should allow people to do whatever they want with the resources, creating new utilities, editing resources, or extracting resources for use in new projects or remakes. I'm hoping easy access to the game resources might spur on new projects.

[Vagabond]

Just finished pushing OP2MapImager code into the repository at: \Outpost2SVN\GameResources\OP2MapImager.

I separated the code into 2 more files: MapFileReader.h and MapData.h. I pushed all the code to parse the map file into a class called MapFileReader. All methods except void ReadMapData(MapData& mapData, string filename) are encapsulated as private. I added documentation to most members of MapData.h since it is a very important public structure. Those comments should be proofed by someone else. I'm still not an expert at the contents of the map file.

I also added helper functions to return the map width in non base 2 log form and the tile set filename as a std::string (the parsed data does not include a '\0' terminator). This way an end user doesn't have to deal with odd implementation details, but the data in the MapData struct still represents the MapFile content.


If anyone wants to continue on the project, I am available on Sunday, 4 June at Midnight GMT - 5 June 3AM GMT. That is Sunday 7-10PM Eastern US Time. If this is too early, we could also push later in the day. Looking at about a 3 hour chunk of time for the pairing.

I'll probably discontinue working on the project until then with the possible exception of researching a library for image manipulation or minor fixes if anyone notices dumb things in the code in the repo over the work week.

I'm happy to work with Hooman again or if someone else wants to jump in, myself or Hooman could rotate out? Or if there are 4 of us, we could pair on 2 different projects.

[Hooman]

Your convenience methods sound like a good idea.

I'd be happy to pair with you again. I've noted it in my calendar.

[Vagabond]

It looks like a console VOL decompress/compressor already exists in the repository (Outpost2SVN\GameResources\VolDecompress). The code looks clean and easy to read. While reviewing it, I realized void main(int argc, char **argv) contains a pointer to a pointer (**)... I had to look that up.

Anyways, the program is incomplete:

Code: [Select]

void OutputUsageMessage()
{
	// Print usage info
	cout << endl;
	cout << "OP2Decomp - The Outpost 2 decompressor/unpacker for VOL and CLM files" << endl;
	cout << endl;
	cout << "To extract all files from an archive:" << endl;
	cout << "OP2Decomp [filename]" << endl;
	cout << endl;
	cout << "To pack a list of files into an archive:" << endl;
	cout << "... not sure on syntax yet (but code exists!)" << endl;
}

It looks like at some point we should probably finish up the console interface of the application and separate the backend logic into a library project. Then the OP2MapImager could just reuse this VOL library instead of re-doing all the VOL decompress code. It looks like a lot of work went into writing the code, including a Huffman Tree for compression. I also had to lookup what a Huffman Tree is.

More generally, I would like new code to allow for easy cross platform extraction of game resources. Many of the current tools are Windows specific, and have extra dependencies that need to be installed. I'd like standalone utilities available with minimal dependencies, to ensure stuff just works wherever. Ultimately, I want to make it easy for anyone to access game resources, either through library code, or without needing to program anything, in easy to use standard formats. That should allow people to do whatever they want with the resources, creating new utilities, editing resources, or extracting resources for use in new projects or remakes. I'm hoping easy access to the game resources might spur on new projects.

Since we are currently working in Visual Studio, I think the code can only be easily compiled for use in Windows. I'm guessing it should be pretty easy to use through Wine on Linux though?

If we keep the applications simple enough and generic, the files could be loaded into a new solution in an IDE that supports a Linux or Macintosh compiler and then compiled for use. Is that sufficient?

I don't know a lot about other compilers, but if we used something else like Vim or EMACs, would that be trivial to switch targets and compile natively for Windows, Linux, or Macintosh. Besides a couple of Code::Blocks projects, I believe everything in the repo right now is Visual Studio.

I'm not well versed on cross platform development.

[Hooman]

Code: [Select]

void main(int argc, char **argv)

The argc is the number (count) of command line arguments. The argv is the actual command line arguments. It is an array of strings, where each string is an array of chars. As each array decays to a pointer, it's simply written as a double pointer.

It might not be a bad idea to finish that project. I'd forgotten about that note about not having a command line interface to repack VOL files.

It's possible the compiled code might run on Linux under Wine. However, the code can't currently be compiled under Linux, which creates a significant barrier to development for anyone using Linux. You could run a compiler in a VM, but this is still an extra step that doesn't integrate well with tooling.

The problem doesn't come from using Visual Studio. That's just an IDE, which is really just a glorified text editor. The problem comes from the code using platform specific features through direct API calls.

The files with platform specific code are CArchiveFile, CVolFile, and CClmFile. Within them, you'll notice "#include <windows.h>". They also make use of the Windows specific HANDLE type, for referencing file handles, along with associated functions: CreateFileA, MoveFileA, DeleteFileA, CreateFileMapping, MapViewOfFile, and WriteFile. To make the code platform independent, those files would need to be rewritten to replace those occurrences.

The files CAdaptHuffTree, CHuffLZ, CBitStream, and even Main.cpp don't appear to use platform specific features themselves, they just rely on the other files that do.

Then there's getting it all to compile in one easy command on a new platform. You'd need a replacement for the Visual Studio project file that basically says what your sources files are, and how to feed them through a compiler. Typically this is done with a Makefile. This is a small issue. Visual Studio already has a built in tool to output a suitable Makefile.

[Vagabond]

@Hooman, I'm still on for the pair programming in about 6 hours (4 June at Midnight GMT - 5 June 3AM GMT). I'll be on Discord and in the OP2 Quake chat room.


I did a little research on cross platform development. This article is about 7-8 years old but I liked some of the advice in it: https://www.backblaze.com/blog/10-rules-for-how-to-write-cross-platform-code/. I also spent a little time reading an overview of makefiles.

The author recommends writing your own functions for file access and other platform specific code and isolate it to a specific areas in your code. You can then control the content of the file access functions using pre-proccessor directives like #ifdef _WIN32. So you could use the native operating system code for each supported operating system in the function. This would isolate all your platform specific code to one or a couple of files and allow you to build with the same .h/.cpp files on any supported platform.

I think a similar option would be to create an interface for the platform specific code functions and then write classes to fulfill the interfaces for each supported platform. You could then use a pre-processor directive to actually load the proper class fulfilling the interface. I have never attempted interfaces in C++, but my understanding is there is no built in support, so you have to sort of hack it.

[Vagabond]

Next session will be this Sunday, 11 June at Midnight GMT - 5 June 3AM GMT (same time as last weekend). That is Sunday 7-10PM Eastern US Time. Looking at about a 3 hour chunk of time. Sorry for the late post.

Looks like the plan will be getting the Outpost 2 specific BMP files loaded into memory, then passing the BMP off to FreeImage for crunching. If time remains, we can work on getting FreeImage to actually scale down an image and paste it into a larger imager.

[Hooman]

Sounds good. Might take a bit of time to handle the custom image format. I should probably review the existing code since it's been a few years.

[Hooman]

@Vagabond: I'm back from my short banishment from the internet. I'll have to look through your updates to see how things sit, though it looks like you've basically got it working.

@leeor_net: I want to know how outposthd development is going, and what your process is like. Now that some patches have been submitted for the code to compile on Linux, there is little excuse not to look more into your project.


On another note, I've been wanting to experiment with Ruby/Rails and websockets. Anyone interested in some exploratory work? I'm thinking of a test project such as a simple chat room, just to see how the technology fits together.

[leeor_net]

I've been focused on another project at the moment, one that brings in a small amount of income but nonetheless is income.

Would def like to continue work on OPHD at some point though.

[Vagabond]

Hooman,

Yes, the OP2MapImager is functioning! Right now I am currently looking through your VOL Decompress code. It looks like I can adopt it without too much trouble to allow the OP2MapImager to search VOL archives in a directory for the given map and wells. I can structure the program to look first in the directory itself and then through the vol files until a map or well with the given name is found. This way a loose file in the directory takes precedent over a file archived in a VOL.

I'll want to modify your VOL decompress code a little to allow sending a list of all filenames within the VOL among other things. How sensitive are you to me mucking around in the VOL decompress code? I would like to make a branch preserving what you have stable. (I've never done this in SVN, so hopefully fairly straightforward).

The other piece is loading the non-standard Outpost 2 TileSet BMPs into memory. I've designed a function within the MapImager class that accepts the raw bits instead of loading a file from the hard drive. I could use some help getting the BMPs into memory. Without help here, I will probably just distribute the modified BMP files with the application instead of hacking my way through the process.

Feel free to work on the MapImager as you see fit. I'm being good about uploading my progress regularly to SVN and checking for updates to keep from getting out of sync with any possible work you do.

Depending on where we want to draw the line on adding new features, the OP2MapImager should be ready for initial release within 2-3 weeks (hopefully?). There is still a lot of cleanup that should be done to the code.

I printed off some talking points about what D is supposed to be and an interview article with one of the 2 creators to read through. It is currently on the TIOBE index as number 22, 1.416% (https://www.tiobe.com/tiobe-index/). Just starting to warm up to trying something in the language and you are already off to Ruby on Rails!

Anyways, I will probably not be available again for pair programming until around middle of August. Once we are in August or my schedule changes earlier then expected, I will see where everyone stands and what the interests are.

[Hooman]

Your precedence rules are correct, in that the game looks for loose files before checking the contents of the VOL files.

I'll want to modify your VOL decompress code a little to allow sending a list of all filenames within the VOL among other things.

Why?

My gut feeling is there's probably a better design to whatever it is you're trying to accomplish, though I'm not clear on what it is you're trying to accomplish.

The other piece is loading the non-standard Outpost 2 TileSet BMPs into memory. I've designed a function within the MapImager class that accepts the raw bits instead of loading a file from the hard drive. I could use some help getting the BMPs into memory. Without help here, I will probably just distribute the modified BMP files with the application instead of hacking my way through the process.

Very practical set of choices here. Not just in code design, but also in project management, and how the code will evolve.


We can cover some D in a future session if you'd like. Ruby as well. Maybe throw in some JavaScript at some point too. I have multiple project ideas. But, all in good time. Let's try to finish what we've started first before getting too carried away.

[Hooman]

It's been a while. I thought I'd make an updated list of project ideas for pair programming sessions.

Vagabond's projects:

• OP2MapImager
• OP2Archive

Vagabond has done an impressive job with these two. They are now largely complete and usable.

A few features can still be finished, such as adding support for the native OP2 tileset in the map imager, and getting the archive code to compile on Linux. It might also be worth exploring the use of Git and GitHub for these projects. There's been some talk of moving coding projects out of our self hosted SVN server, and onto GitHub. This could be a good place to start with that.


Future project ideas:

• Anything from the Needed Future Projects list. These are OP2 related projects that I feel are most important moving forward.
• An OP2 mission: Anyone have any mission ideas, but don't know how to get started? Pair up with someone more experienced to help start you out
• Hacking OP2 Pathfinding: Take a deep dive into the internals of Outpost 2. The pathfinding algorithm and associated data structures has yet to be understood or well documented.
• OutpostHD: Help out leeor_net by working on OutpostHD! This is actually a really important project here. It's come along way, and is one of the few full game projects that looks like it might actually make it. It's getting a lot of interest, and has brought in new users, and convinced a number of lurkers to signup on the forums. I don't think we've had a project that's created quite so much interest and activity before. We need to support this one.
• WebChat: A WebSocks based chat client and server, similar to IRC. This would use JavaScript on the frontend, and possibly use Ruby on Rails for the backend. I view this more as an exploratory project to experiment with and learn some Rails and WebSocks programming. One selfish reason being these skills could look good on a resume. Another reason is simply to increase the web development skills in this community. Though not specifically an objective, people have in the past toyed with ideas such as integrating chat with launching games, or alternate game servers. If someone wants to go down that route, they could use this project as a base.

And remember, any group of people that wants to give pair programming a try is welcome to give it a go. You don't need permission.

[Arklon]

WebChat: A WebSocks based chat client and server, similar to IRC. This would use JavaScript on the frontend, and possibly use Ruby on Rails for the backend. I view this more as an exploratory project to experiment with and learn some Rails and WebSocks programming. One selfish reason being these skills could look good on a resume. Another reason is simply to increase the web development skills in this community. Though not specifically an objective, people have in the past toyed with ideas such as integrating chat with launching games, or alternate game servers. If someone wants to go down that route, they could use this project as a base.

I think a nicer way to go would be making a JS frontend for IRC, which is how Twitch chat works. IRC has stood the test of time, building around that would let people connect to chat using whatever IRC client they want and not be limited to just whatever frontend(s) we provide, more readily enable the use of bots, allow BNCs, etc.

Some Twitch streamers embed chat on their website, which I believe someone had to implement their own JS IRC front end to accomplish that, so presumably there's source code for that around somewhere. If you can find it that'd make a good reference point.

I'd be more interested in the pathfinding stuff as you know, but you also know I'm mostly just interested in procrastination.

[Vagabond]

Actually, I'm available this Sunday 3 Sep at Midnight GMT - 4 Sep 3AM GMT (same time Hooman and I used last time). I can shift this time a few hours later if helpful for someone else.

My first choice would be removing windows.h from the Archive code. If we get far enough, then working on a Linux build either through a makefile or a fresh Linux project. I currently do not have access to a Linux OS so if the pair had Linux it would be helpful! However it might take the full time to get windows.h out of the CLM code, so no big deal if we just work Windows for the session. I would tend to test code changes using the OP2Archive program since it has a working EXTRACT and CREATE command.

Other things I would be interested in is hooking Google Test into OP2Archive for unit testing. I'm not looking to commit to a major project right now (especially since OP2Archive isn't complete) though I suppose I could help for the session if someone has a burning desire to start something bigger.

Sorry for the short notice.

-Brett

[Hooman]

I think a nicer way to go would be making a JS frontend for IRC, which is how Twitch chat works. IRC has stood the test of time, building around that would let people connect to chat using whatever IRC client they want and not be limited to just whatever frontend(s) we provide, more readily enable the use of bots, allow BNCs, etc.

To be clear, I'm not proposing we replace IRC. You're right in that IRC has withstood the test of time.

I'm suggesting this more as a potentially fun simple small project to start increasing the amount of web development skills around here. If someone wants to take the code and run off to build some crazy hair brained scheme, they're more than welcome, though I have no plans on using this project to replace IRC. It's also why I don't want to use pre-built code.

As for having a JavaScript frontend that interfaces with IRC, I don't believe that's possible without going through a custom server. One of the aspects of WebSockets, is you have no control over the wire representation of your data. This is for security reasons. The browser will send each packet XORed with a random mask (included in the header of the packet), which the server must undo before processing the data. This value changes with each packet, using a secure random number generator, to ensure the server doesn't communicate the mask back to the client and have the client trivially undo the masking. This is specifically to prevent browser traffic from impersonating traffic for other protocols. Remember browsers are running untrusted code from over the internet, so the implications of being able to write an IRC client in the browser would be great.

Basically what I'm saying is, any IRC like JavaScript frontend, will need to communicate with a custom server, perhaps written using Rails (or Node, or whatever). Whether the server then interfaces with IRC is a separate question.

I'd be more interested in the pathfinding stuff as you know, but you also know I'm mostly just interested in procrastination.

I included that one for a reason.    Now would you like to schedule a time to procrastinate, or perhaps procrastinate about procrastinating?

Actually, I'm available this Sunday 3 Sep at Midnight GMT - 4 Sep 3AM GMT

That's 4 Sep 9am - noon my time, so that would work for me. I made a note on my calendar to be on IRC at that time.

My first choice would be removing windows.h from the Archive code. If we get far enough, then working on a Linux build either through a makefile or a fresh Linux project. I currently do not have access to a Linux OS so if the pair had Linux it would be helpful! However it might take the full time to get windows.h out of the CLM code, so no big deal if we just work Windows for the session. I would tend to test code changes using the OP2Archive program since it has a working EXTRACT and CREATE command.

Agreed, removing windows.h from the Archive code is a priority. I was taking a look at my old VolDecompress project last night, testing some changes in a Windows VM. I figured I'd start there since, well, it was my old project and I'm most familiar with it. Plus I wouldn't be stepping on your toes if you were doing active development on your project. I should probably switch over to your project though. At the very least, changes should be directly applicable to what you're doing. Perhaps following a set of small changes, using Git, could be a good way to become familiar with that version control system.

I think once we have the windows.h stuff removed, then we can switch to developing on Linux. I can host for that. That might be a future session though.

The Google Test stuff could be interesting. For now I'm considering that a problem for a future session.

[Vagabond]

Agreed, removing windows.h from the Archive code is a priority. I was taking a look at my old VolDecompress project last night, testing some changes in a Windows VM. I figured I'd start there since, well, it was my old project and I'm most familiar with it. Plus I wouldn't be stepping on your toes if you were doing active development on your project. I should probably switch over to your project though. At the very least, changes should be directly applicable to what you're doing. Perhaps following a set of small changes, using Git, could be a good way to become familiar with that version control system.

I think once we have the windows.h stuff removed, then we can switch to developing on Linux. I can host for that. That might be a future session though.

Sounds good. I haven't made drastic modifications to the decompress code so any ideas you have will be easy to transfer over. I'll be on IRC as well. I'll plan to update to the newest version of Team Viewer, Discord, and my IRC client Friday or Saturday so they won't be pestering me to update during the pair programming.

-Brett

[Arklon]

To be clear, I'm not proposing we replace IRC. You're right in that IRC has withstood the test of time.

I'm suggesting this more as a potentially fun simple small project to start increasing the amount of web development skills around here. If someone wants to take the code and run off to build some crazy hair brained scheme, they're more than welcome, though I have no plans on using this project to replace IRC. It's also why I don't want to use pre-built code.

As for having a JavaScript frontend that interfaces with IRC, I don't believe that's possible without going through a custom server. One of the aspects of WebSockets, is you have no control over the wire representation of your data. This is for security reasons. The browser will send each packet XORed with a random mask (included in the header of the packet), which the server must undo before processing the data. This value changes with each packet, using a secure random number generator, to ensure the server doesn't communicate the mask back to the client and have the client trivially undo the masking. This is specifically to prevent browser traffic from impersonating traffic for other protocols. Remember browsers are running untrusted code from over the internet, so the implications of being able to write an IRC client in the browser would be great.

Basically what I'm saying is, any IRC like JavaScript frontend, will need to communicate with a custom server, perhaps written using Rails (or Node, or whatever). Whether the server then interfaces with IRC is a separate question.

True. Twitch's IRC server is custom, and so are their services. Luckily there's lots of good open source IRCd's that could just be modified to be websocket-friendly. We used InspIRCd back when we ran our own server, which is pretty good as I recall and it's open source. Not sure what good services packages there are out there; we used Anope and Epona with our server I think but those were questionable. For something like forum integration we'd have to roll our own NickServ though.

[Hooman]

Forum integration, now there's an idea! 


As for the pair programming sessions, we had a 3-way last time, so no need to limit to just 2 people per session. It seems TeamViewer is quite capable of connecting more people at the same time.

[leeor_net]

That's because TeamViewer is awesome and I love them for allowing free use for personal use. I wish I could afford to pay them to support their development.

[Vagabond]

We had another successful pair programming yesterday between myself and Hooman. It was more exploratory than earlier sessions in which we had a specific goal in mind.

We hooked OP2Utility to both OP2Archive and OP2MapImager via submodules in Git. We then spent some time playing around with commits and pushes in the submodule to see how it propagated between the applications depending on the submodule (OP2Utility).

VS2017 has built in integration with Git. This is nice, because you can see at a glance which files in your solution explorer have uncommitted changes in them. We were able to push Git repo changes directly from VS2017. We spent some time exploring Git in VS2017. I think I still prefer the interface of using TortoiseGit more than doing it from VS2017, but we didn't spend enough time there to make that call for sure.

The last piece left undone was figuring out how to update the version of the submodule a repository uses when submodule changes are pushed.

If others are interested in pair programming in any form, I would encourage them to post here with times and projects. Don't feel like you need a specific amount of expertise to participate.

And I'll echo Leeor that TeamViewer has been a great application for the pairings.

-Brett

[leeor_net]

As a note, VS2015 also has Git built in.

Anyway, would love a rundown of the submodules thingy you guys were working with. It's something I've been meaning to look at but never seem to have the time.

[Vagabond]

I wanted to schedule another pair programming session for this coming weekend to work on OP2Ext.

If someone wants to move OP2Ext over to Github beforehand, that would be nice or we can start the session out with that. After moving it over, I'd like to tag the current code state and place the current DLL in the release section of GitHub created by the tag. I guess we could call it version 1.0.0 unless someone has a better idea. After that, We will want to work on getting it compiling with a newer version of Visual Studio (VS) or through a makefile first then VS.

I can do Sunday from 3 Sep at Midnight GMT to 4 Sep 3AM GMT. (Same time Hooman and I have worked last several times). I can push later in the day 4 SEP if that is better for someone else. I'm also available some on Saturday if that works better than Sunday.

Anyway, would love a rundown of the submodules thingy you guys were working with. It's something I've been meaning to look at but never seem to have the time.

Leeor, I haven't forgotten about this. I'll try to make some time perhaps in January if your interest holds. It isn't too difficult when you start doing it regularly.

-Brett

[Hooman]

Sounds like a plan.

... though you might want to check the Calendar

I assume you mean this time:
https://www.worldtimebuddy.com/?qm=1&lid=0,1668341,1668341&h=0&date=2017-11-13&sln=0-3

I'm guessing the project has yet to be compiled under a current version of Visual Studio.

[Vagabond]

Hooman,

Sorry, you are correct. Sunday, 12 Nov midnight GMT to 13 Nov 3am GMT.

No I have not started porting it to a neweer version of Visual Studio. I was wanting to wait until the current code base was pushed to Git. I may have some time to try to Port beforehand though.

Brett