Moving Pictures

[RobbixTheBobbix]

Project of mine I've been working on for a few months.

Youtube clip of demonstration

Google Code project where you can download the work in progress

Requires Java 1.6, might run in 1.5, haven't tried it. It's packaged on the Google Code page as an Eclipse project, so you can import it and have your way with it. Runs on windows, should run on linux and mac.

The video was recorded on my circa 2005 laptop where I'm doing the programming and it can run smoothly and a couple times faster than the nominal speed, despite Java. When I tried it at work, it ran at about 90% the nominal speed and was sketchy despite it being a more powerful circa 2008 machine.

Results may vary.

The interface will be kind of awkward until I get all the command buttons working - they're a fairly new addition.

* Left click units to select them.
* Left click a space to give a move command.
* Right click to deselect or cancel a unit placement command.
* Middle click the red command names on the edge of the screen to give commands.

uh....

Questions?

-----------------------------------------------
PS:

I don't know if there's going to much excitement about this as I've read about the repeated, crippling disappointments that have come from op2 clone projects in the past. I intentionally didn't post about it until I had gotten somewhere.

I have a habit of working on proofs at concept at work over lunch or just whenever I don't feel like working. I have PoCs concerning

* Map scrolling (current maps are only the size of the screen)
* Terrain editing and automatic tile annotation (related to map generation where tiles that go together have to be automatically selected)
* Tube connectivity <- works like a charm
* Networking (which is actually far-off, but I'm trying to think ahead structurally)
* More advanced pathfinding and automatic map annotation
* There are some others, I'm not at work.

--------------------------------------------------
PPS - About contributing:

So far, I've been extracting animation sequences from the op2art program into individual files - one frame per file - and writing info.xml files that contain information about them.

This is very tedious.

It also runs slow as the program has to load and read a couple 1000 individual files each time it starts.

So I might be asking for help with do the extracting and writing the info.xml files or I might be asking for help in loading the original op2_art.bmp/op2.prt files. I designed Moving Pictures to be modular enough that changing the sprite load sequence requires only the editing of a few lines. (Well, and the addition of a 1000 more for the actual load code, but whatever...)

I have considered writing my own binary file format to store each unit's sprite set and info about as I like the idea of modular sprite loading, which MP uses, but performance concerns at load time may make me go the op2_art.bmp route.

Oh, and please don't inundate me with feature requests. I know all about the features of the game and I'm keeping in mind things like variations on multiplayer, extensions to trading, around the world maps, tilesets, new units, etc. Right now, suggestions on more fundamental elements of the engine would be preferred, if anyone wishes to offer any.

-------------------------------------------------------------------------------------
Addendum: Graphics

Right now, the game's graphics kind of awkwardly relies on Java's built-in Event Dispatch Thread which is used to paint normal UI components like buttons and text fields and handle input events. And since the mechanics, which includes unit and environment behaviors, since mechanics also runs in the EDT, the program is primarily single-threaded. That's why, if sprites are set to load on demand, when you create a new type of unit (especially the cargo truck), the program will hang while the files are loaded, which ALSO occurs in the EDT.

Just some insight into how it works.

I tried implemented an asynchronous approach where mech and graphics and loading all run in different threads but it didn't run as smoothly as it does now.

---------------------------------------------------------------------------
Addendum: Audio

I found it most convenient to use javax.sound.sampled.Clip objects for the sound bytes since they have a simple start() stop() interface and mix together. However, each Clip instance has it's own thread associated with it, so when all the sounds are loaded the program is running in a couple hundred threads.

On top of that, I decided to include loading sounds modularity on demand, just like sprites and unit types, so the first time the meteor hits the ground, it has to load the sound and a bunch of sprites for the death of the units it strikes. Unchecking "load ____ on demand" when launching the sandbox might help to prevent this, although it varies by machine and Java version.

I'll work on more well-thought-out version of the graphics and audio, but I don't think I'll be jumping for native libraries as that would start to defeat one of the only benefits of using Java for a game. (for a GAME of all things!)

[Savant_Ace]

Looks quite promising, but you are correct, it won't catalyze much excitement based on all the new project blue balls that have been thrown around in the past.

Still, awesome! B)  

[Hooman]

Based on the YouTube video, this is far more than I've seen posted of other attempts. I'm quite pleased to see this.

[BlackBox]

I watched the youtube video and was pretty impressed. I haven't looked at the code yet but it seems potentially promising (if there are major guts of an RTS engine here).

It might not be terribly difficult to improve performance (I don't know if we would want to use OP2's original formats necessarily, as opposed to writing converters for them and using some format that is easier to work with using existing tools.. i.e. loading png images from zip files or something. I'm also assuming it uses java2d for graphics, that might not be as performant as other potential approaches like SDL or similar). I'm an enterprise software developer by day (including java, and we have to write for memory/performance constrained platforms such as smartphones) so I would definitely be willing to lend my assistance, I just don't know how much time I could devote.

If it seemed like this project has a good chance of making it to fruition (depends on who is contributing and how much people are able to realistically contribute due to time constraints) we could always try to get the community involved. It wouldn't be hard for us to add sections to the OPU site and so forth, if people were interested.

[evecolonycamander]

that looks good. very much so. It reminds me of something i tried in school once with game maker. it didn't get far though. not like your project that seems to be a wow moment
 

[darwithe]

I've got to admit, I'm very impressed.

Few things, first it doesn't look like would it really require much more than changing the XML, and XML reader slightly to make it take animation strips instead of completely individual frame files. That would let you reduce the loading from the current 8,000 3000ish files down to 1,000 100ish while retaining the vast majority of the useful modularity.

Next, the image extraction/infoization looks simple enough, thou granted the tedium, that I'm willing to help. Any area in particular you want me to focus, or should I just pick something not included in the release and start chugging away?

Lastly, while these probably don't qualify as 'Fundamental' they do need your design to keep from making assumptions to keep from being a pain to add, so I would like the ability to break down resources into finer chunks (IE: Iron, Copper, Tin, Etc from commons, Iridium et all from rares), create additional resource sources, and create additional damage types which effect specific extra health (IE: Viral weapon that does infection type damage, which doesn't hurt normal health but only a custom viral HP. For that matter EMP could be handled in a similar fashion.)

Edit: SVN folders inflated original file counts.

[Spikerocks101]

Hello Darwithe, I dont recall seeing you before. I don't think RobbixTheBobbix is tacking suggestions atm, since he/she is on a role with doing it them self.

[Highlander]

Based on the YouTube video, this is far more than I've seen posted of other attempts. I'm quite pleased to see this.

You should see _A_'s Demo

I don't think she has posted it anywhere though - but it is playable

[darwithe]

Hello Darwithe, I don't recall seeing you before. I don't think RobbixTheBobbix is taking suggestions atm, since he/she is on a role with doing it them self.

I only register on a forum very reluctantly, so I've never posted here before, but I've been an intermittent visitor since 2004 or so when I dug out my OP2 CD for the umpteenth time and suddenly wondered if there were any unofficial bug-fixes or whatnot.

So yeah, 'Hi!'

As for the suggestions: RobbixTheBobbix mentions in one of his edits that "Right now, suggestions on more fundamental elements of the engine would be preferred, if anyone wishes to offer any."

[RobbixTheBobbix]

1) Few things, first it doesn't look like would it really require much more than changing the XML, and XML reader slightly to make it take animation strips instead of completely individual frame files.

2) Next, the image extraction/infoization looks simple enough, thou granted the tedium, that I'm willing to help. Any area in particular you want me to focus, or should I just pick something not included in the release and start chugging away?

3) I would like the ability to break down resources into finer chunks (IE: Iron, Copper, Tin, Etc from commons, Iridium et all from rares), create additional resource sources, and create additional damage types which effect specific extra health (IE: Viral weapon that does infection type damage, which doesn't hurt normal health but only a custom viral HP. For that matter EMP could be handled in a similar fashion.)

(*addendum added to OP*)

1) I was considering doing something like this as an automated process that would take a huge folder like "eCargoTruck" and lump all the info and bmps into a single "eCargoTruck.dat" file. This would effectively prevent the game from having to redo the same work of accessing all those files every time.

2) If you want to, you could look at units like "eCargoTruck" and build a mirror-imaged folder like "pCargoTruck" (plymouth cargo truck). If the units are the same except for coloring, then you won't have to go through the tedium of copying out the x,y offsets. Try something like that first, then go for "pVehicleFactory", "eRareSmelter", building the folders just like their Eden cousins. You can test it by putting the folder in with your downloaded copy of the game (/res/sprites) to try it out and zip/send me the results.

Oh yeah, you'll also have to make unit definition files for /res/units.

3) Those wouldn't be too hard to add, don't worry about it. But I'll keep it in mind.

[RobbixTheBobbix]

You should see _A_'s Demo

I don't think she has posted it anywhere though - but it is playable

Any idea where it is?

[TH300]

This looks pretty nice.

Here are some suggestions (some of them have are not new):

Don't use that many files. Not only do they kill performance, they also take up more space on the filesystem than few big files. If you are interested I can contribute code to load tileset images from original op2 files.

Use opu git: http://forum.outpost2.net/index.php?showtopic=5159 - this will make it easier for opu coders to contribute. I am generally interested in contributing if you need help. First I have to get familiar with the code though, which btw. looks really neat and clean.

Use a different solution for sound playback. Spawning a few meteors will almost freeze the whole application for me if sounds are on. If sounds are off it runs fluidly. You could use OpenAL, for example.

You should see _A_'s Demo

I don't think she has posted it anywhere though - but it is playable

Any idea where it is?

Its not public, yet. Hence I doubt you'll find it somewhere. Despite _A_'s remake seems to be pretty complete, work on your one won't be useless, since _A_'s is not open source and not platform independent.

[RobbixTheBobbix]

First I have to get familiar with the code though, which btw. looks really neat and clean.

Use a different solution for sound playback. Spawning a few meteors will almost freeze the whole application for me if sounds are on. If sounds are off it runs fluidly. You could use OpenAL, for example.

Parts of the code are definitely not neat and clean. Some parts were hack together just to get a feature in there and weren't implemented in a very elegant way.

Which brings me to the audio...

I found it most convenient to use javax.sound.sampled.Clip objects for the sound bytes since they have a simple start() stop() interface and mix together. However, each Clip instance has it's own thread associated with it, so when all the sounds are loaded the program is running in a couple hundred threads.

On top of that, I decided to include loading sounds modularity on demand, just like sprites and unit types, so the first time the meteor hits the ground, it has to load the sound and a bunch of sprites for the death of the units it strikes. Unchecking "load ____ on demand" when launching the sandbox might help to prevent this, although it varies by machine and Java version.

I'll work on more well-thought-out version of the graphics and audio, but I don't think I'll be jumping for native libraries as that would start to defeat one of the only benefits of using Java for a game. (for a GAME of all things!)

[TH300]

When disabling sound loading on demand I get

Code: [Select]

java.io.IOException: javax.sound.sampled.LineUnavailableException: line with format PCM_UNSIGNED 22050.0 Hz, 8 bit, mono, 1 bytes/frame,  not supported.
        at com.robbix.mp5.ui.SoundBank.preload(SoundBank.java:48)
        at com.robbix.mp5.Game.load(Game.java:38)

You should rather find a solution that keeps all sounds in memory and doesn't use one thread for every sound.

I never programmed anything with audio, so I can help you little with that.

I suggested OpenAL, because its one of the (probably few) sound libraries with java bindings (namely JOAL). Its probably the most platform independent sound library. I don't know if it is suitable for your game. But its used by some serious games, so its probably worth being looked at.

[RobbixTheBobbix]

When disabling sound loading on demand I get

Code: [Select]

java.io.IOException: javax.sound.sampled.LineUnavailableException: line with format PCM_UNSIGNED 22050.0 Hz, 8 bit, mono, 1 bytes/frame,  not supported.
        at com.robbix.mp5.ui.SoundBank.preload(SoundBank.java:48)
        at com.robbix.mp5.Game.load(Game.java:38)

Yeah, the JavaSound package acts differently on different platforms. Yours doesn't support the format the sound bites are in. It always worked on mine, so I never noticed. Sorry.

[BlackBox]

Same thing happened on Mac OS X 10.6.8 with apple's JVM. (Though mine was somewhat less descriptive, the description was "no free voices" on the exception it threw).

I'd suggest using something like SDL, it's an extremely cross platform graphics/sound/input handing library. There are JNI bindings to it (SDL itself is written in C and is very portable).

As far as dealing with the slowness in reading files, you should definitely try to avoid the huge number of opens you do (one simple idea might be to use a ZipInputStream). If you are using something like SDL for example you can deal with loading bitmaps and drawing a lot more efficiently as well (since the bitmaps can be loaded from the native side into memory buffers, so there isn't all sorts of marshalling that continually needs to happen between the java and native side).

For metadata storage XML parsing is going to be extremely slow with the amount of data you're reading (XML is not an efficient format to parse in the first place). One possible idea might be some kind of database (SQlite comes to mind, it is also native C but is portable anywhere and there are also java bindings for it. If you wanted pure Java you could try something like HSQLDB as well). There are tools to work with databases created by these libraries, so you don't need to write tools to handle creation/maintenance of them.

[Hooman]

Based on the YouTube video, this is far more than I've seen posted of other attempts. I'm quite pleased to see this.

You should see _A_'s Demo

I don't think she has posted it anywhere though - but it is playable

I qualified that with "posted" for a reason.


I believe there can be a less manual way of extracting the graphics. We do know a reasonable about about the file formats, and do have code written that's been able to parse and display it. I believe there is also a list of indexes for the unit animations that can potentially be used to extract all of those. The user interface animations and graphics would probably be the harder ones to extract and sort nicely. Those would likely require more manual work. The unit graphics though can probably be done in an automated way.

I will also agree with the others that have suggested lumping the graphics into fewer larger files. Personally, I think one file for all graphics will do. I find BlackBox's suggestion about the zip file curious, and strangely practical. (It seemed like the kind of solution I would expect in a workplace environment). It should solve the issues with the number of open calls and the filesystem size overhead, and shouldn't require extensive code changes or data format changes. I suspect working on that could quite drastically reduce your startup time.

[evecolonycamander]

I think I have to say it. No other project(with in the time i have been here) has created such an uproar in this manor. kudos to you. what makes this even more amazing is that you did this as a one man team. few get as far as you in anything like this(despite historians glorifying the 'lone wolf' status.) if this continues, as i surely hope it will, i foresee us finally getting that magical op2 remake with included source.

sorry for the irrelevancy of this post, but i had to say it.

[TH300]

I like the idea of using zip archives as container for several small image files. When doing a little test with the Eden SF "fidget1" frames I get better compression than converting each file to png.

As for the choice xml or not, I can tell that its probably a bottleneck, but at a place where it won't hurt. You have to parse xml files only once at startup (either of the whole application or a single game) and that is still fast enough to be unnoticed. (Civilization IV uses xml and it has no performance issues)

I like the xml format, because it can be modified in a mere text editor.

One thing that I don't like is its verbosity, especially the lines

Code: [Select]

  <OffsetFrame offsetX="11" offsetY="41"/>
  <OffsetFrame offsetX="11" offsetY="41"/>
  ...

We should find a better solution for those.

[Hooman]

Sadly, I think that's probably the more compact way to express that in XML. Be thankful it's not something like SLD style files used in map styling for various geographic information systems. I've always found them to be needlessly verbose, even for XML. They seem to not like tag attributes. I think their equivalent would have been more like:

Code: [Select]

<OffsetFrame>
  <Parameter>
    <OffsetX>11</OffsetX>
  </Parameter>
  <Parameter>
    <OffsetY>11</OffsetY>
  </Parameter>
</OffsetFrame>

Those files have always made me cringe.
 

[RobbixTheBobbix]

I like the idea of using zip archives as container for several small image files. When doing a little test with the Eden SF "fidget1" frames I get better compression than converting each file to png.

I like the xml format, because it can be modified in a mere text editor.

One thing that I don't like is its verbosity, especially the lines

Code: [Select]

  <OffsetFrame offsetX="11" offsetY="41"/>
  <OffsetFrame offsetX="11" offsetY="41"/>
  ...

We should find a better solution for those.

Yeah, the frm-xxxx.bmp files compress fairly well, as there is alot of repetition in them. It's the sound files that make the download so big. And all the unused sound files are in there too.

I was thinking of re-writing them using JSON or YAML. In JSON, that XML sample would could look like this:

Code: [Select]

  ["11", "41"],
  ["11", "41"],
  ...

I don't want to use one big file as I like the idea of modularity. I have this overly ambitious idea that we could be able to pipe things like new maps, new sprite sets, new sounds, new tilesets, even new unit types over the network on the fly as a multiplayer game is loading.

I think I'll put it on the list to have the sprite loader read two formats:
* open set of folders and files with a xml config file
* a single binary file that contains the config info and all sprites for a unit/event. May also be zipped.

And an extra program that builds the .dat file. That way, it can be uncompressed for editing, and then compressed for deployment.

[Highlander]

You should see _A_'s Demo

I don't think she has posted it anywhere though - but it is playable

Any idea where it is?

Its not public, yet. Hence I doubt you'll find it somewhere. Despite _A_'s remake seems to be pretty complete, work on your one won't be useless, since _A_'s is not open source and not platform independent.

Hmm, I'm not one for technical stuff, but according to _A_ her game is cross platform. I'm sure you can ask her on IRC for details if you want/need them - I'm content just to play the demos shes putting out




So far her game seems be coming along quite well, alltough as she says herself, it goes forward in her own time. Last time I spoke to her, she seemed to need some more testers even

Though I have to say I'm a bit puzzled her project has not recieved any sort of attention here. Most people seem aware it exists. Not to mention many of the guys on IRC seems to be using _A_ as a "Go to Guy" or perhaps I should say "Go to Gal" when it comes to coding questions.


Oh well, sorry for sort of hijacking a thread. I'm very glad people still choose to work on this ancient game. Hopefully your project will be fruitful Robbix!

[Spikerocks101]

I don't think this is the only/official thread to this project, since it seems alot bigger then 1 thread. As for _A_, let her introduce her project in her own time. I doubt she wants us to hype up her project while shes not even looking. And Robbix, what are you wanting done with this project? BB already stated that if you need stuff from the website, he can give it to you. Im pretty useless, so I just talk.

[TH300]

I was thinking of re-writing them using JSON or YAML. In JSON, that XML sample would could look like this:

Code: [Select]

  ["11", "41"],
  ["11", "41"],
  ...

Sometimes my ideas are a bit ugly... Judge for yourself:

Keep the xml format for the files

Introduce a new data type "matrix" that looks like this: "[11,41;11,41;...]". It represnts

Code: [Select]

11 41
11 41

Files could then contain lines like

Code: [Select]

<OffsetFrames matrix = "[11,41;11,41;...]" />

Handle this in your program by reading it as a string and parsing it with your own function.
 

[RobbixTheBobbix]

Map scrolling added so now you can have maps bigger than the screen!!! This was an obvious lack of feature which actually only took about 20 lines of code to add, because of some planning-ahead I exercised earlier.

Youtube video of combat and mining on 48x48 map. Before I could only fit a 30x20 map on my 1024x768 laptop screen, so not much could fit.

The combat demo now has four groups of tanks numbering 150 each (mw, laser, railgun, rpg) converging on the center of the map where 81 acid clouds await. Running it reveals serious performance problems with all those vehicles on screen.

The mining demo isn't much different, just more smelters, mines and trucks. This one hardly slows down at all from the bigger map - a pleasant surprise.

So... considering the noted performance problems, and the fact that the combat demo ends up using about 150MB of memory, I would really appreciate a code review. The google code project is linked at the top of the page. It can be checked out through subversion or browsed on the site.

Memory-wise: there are literally hundreds of thousands of Position objects being created as the program runs. The Position class contains only two integer values: x and y. Because there's no way to allocate the pair on the stack in Java, they get allocated on the heap and sit until the garbage collector cleans them up - any ideas? The profiler also showed a similar number of HashMap entries - so that's something else to look for. Look at

Code: [Select]

com.robbix.mp5.basics.Position

and

Code: [Select]

com.robbix.mp5.ui.SpriteLibrary

.

CPU-wise: I ignored this other problem in an attempt to avoid premature optimization, but its time has come: Every mechanics cycle, the entire list of units on the map is copied and iterated over. Every draw cycle (immediately following a draw cycle) the entire list of units is copied (sorted) to a sorted set and iterated over. I suspect this takes some time for 100's of units. Look at

Code: [Select]

com.robbix.mp5.Engine

and

Code: [Select]

com.robbix.mp5.ui.DisplayPanel

.