Author Topic: Linux Data Folder Issue  (Read 6231 times)

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4954
Linux Data Folder Issue
« on: October 01, 2017, 07:05:22 PM »
Although the code was made to compile cleanly on Linux, there is still a runtime issue finding assets.

Quote
$ ./OPHD
Initializing Filesystem... done.
File 'fonts.dat' does not exist as specified.
File 'maps.dat' does not exist as specified.
File 'planets.dat' does not exist as specified.
File 'robots.dat' does not exist as specified.
File 'sfx.dat' does not exist as specified.
File 'structures.dat' does not exist as specified.
File 'sys.dat' does not exist as specified.
File 'ui.dat' does not exist as specified.
Initializing Configuration... done.
Initializing Mixer... done.
Starting OpenGL Renderer:
   - OpenGL System Info -
   Vendor: Intel Open Source Technology Center
   Renderer: Mesa DRI Intel(R) Bay Trail
   Driver Version: 3.0 Mesa 17.0.7
   GLSL Version: 1.30
Unable to load 'ui/pointers/normal.png'. No such file or directory.
OGL_Renderer::addCursor(): 'OpenGL Renderer' is empty.
Unable to load 'ui/pointers/place_tile.png'. No such file or directory.
OGL_Renderer::addCursor(): 'OpenGL Renderer' is empty.
Unable to load 'ui/pointers/inspect.png'. No such file or directory.
OGL_Renderer::addCursor(): 'OpenGL Renderer' is empty.
Unable to load 'sfx/cc_no_bulldoze_m.ogg'. No such file or directory.
Unable to load 'sfx/insufficient_resources_m.ogg'. No such file or directory.
Unable to load 'sfx/invalid_digger_placement_m.ogg'. No such file or directory.
Unable to load 'sfx/invalid_miner_placement_m.ogg'. No such file or directory.
Unable to load 'sfx/invalid_structure_placement_m.ogg'. No such file or directory.
Unable to load 'sfx/invalid_tube_placement_m.ogg'. No such file or directory.
Unable to load 'sfx/ni.ogg'. No such file or directory.
Unable to load 'sfx/location_not_bulldozed_m.ogg'. No such file or directory.
Unable to load 'sfx/max_dig_depth_m.ogg'. No such file or directory.
Unable to load 'sfx/no_tube_connection_m.ogg'. No such file or directory.
Unable to load 'sfx/ni.ogg'. No such file or directory.
Unable to load 'sfx/unsuitable_landing_site_m.ogg'. No such file or directory.
Unable to load 'sfx/ni.ogg'. No such file or directory.
Unable to load 'sfx/ni.ogg'. No such file or directory.
Unable to load 'sfx/cc_no_bulldoze_f.ogg'. No such file or directory.
Unable to load 'sfx/insufficient_resources_f.ogg'. No such file or directory.
Unable to load 'sfx/invalid_digger_placement_f.ogg'. No such file or directory.
Unable to load 'sfx/invalid_miner_placement_f.ogg'. No such file or directory.
Unable to load 'sfx/invalid_structure_placement_f.ogg'. No such file or directory.
Unable to load 'sfx/invalid_tube_placement_f.ogg'. No such file or directory.
Unable to load 'sfx/ni.ogg'. No such file or directory.
Unable to load 'sfx/location_not_bulldozed_f.ogg'. No such file or directory.
Unable to load 'sfx/max_dig_depth_f.ogg'. No such file or directory.
Unable to load 'sfx/no_tube_connection_f.ogg'. No such file or directory.
Unable to load 'sfx/ni.ogg'. No such file or directory.
Unable to load 'sfx/unsuitable_landing_site_f.ogg'. No such file or directory.
Unable to load 'sfx/ni.ogg'. No such file or directory.
Unable to load 'sfx/ni.ogg'. No such file or directory.
Unable to load 'sys/lairworks-logo.png'. No such file or directory.
Image::load(): 'sys/lairworks-logo.png' is empty.
Unable to load 'sys/nas2d_logo.png'. No such file or directory.
Image::load(): 'sys/nas2d_logo.png' is empty.
Unable to load 'sys/ophd_large.png'. No such file or directory.
Image::load(): 'sys/ophd_large.png' is empty.
Unable to load 'music/splash.ogg'. No such file or directory.
Unable to load 'fonts/opensans.ttf'. No such file or directory.
Unable to load 'fonts/opensans-bold.ttf'. No such file or directory.
Unable to load 'fonts/opensans.ttf'. No such file or directory.
Unable to load 'sys/bg1.png'. No such file or directory.
Image::load(): 'sys/bg1.png' is empty.
Unable to load 'sys/flare_1.png'. No such file or directory.
Image::load(): 'sys/flare_1.png' is empty.
Unable to load 'sys/flare_2.png'. No such file or directory.
Image::load(): 'sys/flare_2.png' is empty.
Unable to load 'sys/flare_3.png'. No such file or directory.
Image::load(): 'sys/flare_3.png' is empty.
Unable to load 'sys/cloud_1.png'. No such file or directory.
Image::load(): 'sys/cloud_1.png' is empty.
Unable to load 'sys/cloud_2.png'. No such file or directory.
Image::load(): 'sys/cloud_2.png' is empty.
Unable to load 'music/menu.ogg'. No such file or directory.
Unable to load 'sfx/click.ogg'. No such file or directory.
Unable to load 'sfx/menu4.ogg'. No such file or directory.
Unable to load 'ui/skin/button_top_left.png'. No such file or directory.
Image::load(): 'ui/skin/button_top_left.png' is empty.
Font 'fonts/opensans.ttf_10pt' was not found in the resource management.
Font 'fonts/opensans-bold.ttf_14pt' was not found in the resource management.
Font 'fonts/opensans.ttf_14pt' was not found in the resource management.
Attempted to copy an Image in an invalid load state.

I believe this is related to the platform guard found in NAS2D in src/Filesystem.cpp, in method:
Code: [Select]
void Filesystem::init(const std::string& argv_0, const std::string& startPath)

Simply removing the #if/#else/#endif guards allows the game to work. Though there are still error messages about the fonts:
Quote
$ ./OPHD
Initializing Filesystem... done.
File 'fonts.dat' does not exist as specified.
File 'maps.dat' does not exist as specified.
File 'planets.dat' does not exist as specified.
File 'robots.dat' does not exist as specified.
File 'sfx.dat' does not exist as specified.
File 'structures.dat' does not exist as specified.
File 'sys.dat' does not exist as specified.
File 'ui.dat' does not exist as specified.
Initializing Configuration... done.
Initializing Mixer... done.
Starting OpenGL Renderer:
   - OpenGL System Info -
   Vendor: Intel Open Source Technology Center
   Renderer: Mesa DRI Intel(R) Bay Trail
   Driver Version: 3.0 Mesa 17.0.7
   GLSL Version: 1.30

This is also the function that contains the embedded ".lom" string, which probably shouldn't be in NAS2D. It might make more sense to either pass in that string, or to somehow base it off the executable filename.


Also, while testing this, I found another more recent error in the outposthd repository. In src/ProductPool.h, I needed to add some extra braces to get this line to compile:
Code: [Select]
ProductTypeCount        mProducts = {{ 0 }};

Offline lhark

  • Newbie
  • *
  • Posts: 11
Re: Linux Data Folder Issue
« Reply #1 on: October 02, 2017, 11:56:59 AM »
Apparently, on linux, OPHD/NAS2D searches for config files and assets in ~/.lom, but the files aren't automatically copied there.

One workaround is to manually copy them. A better solution would be to either automatically copy the files or use the same layout as in windows with the config files and assets sitting alongside the rest of the installation.

Offline leeor_net

  • Administrator
  • Hero Member
  • *****
  • Posts: 2350
  • OPHD Lead Developer
    • LairWorks Entertainment
Re: Linux Data Folder Issue
« Reply #2 on: October 02, 2017, 04:58:31 PM »
Assets and config sitting alongside the binary makes sense though because the config file needs to be writable I'm not sure how well that would work on Linux.

The big issue is save games -- it will need a write directory to be able to save, something that the game can create when needed.

As for the *.dat files, the game is looking for those to mount -- but they don't exist yet. Later on they'll be ZIP files with a .dat extension. Will clear up those issues.

As for the .lom string, that's legacy code from when NAS2D wasn't NAS2D yet. It was part of a project called The Legend of Mazzeroth so it made sense at the time to have that hard coded. Now, of course, it makes no sense.

Easiest solution would be to add a parameter to the end of the list called const std::string& userDirectory, give it a default value of "" and have the code test for an empty string. If not empty, create the user directory and use that for writing (create file/directory functions should be updated to check for this). Doesn't break any existing code and allows it to operate properly on pretty much all modern operating systems.

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4954
Re: Linux Data Folder Issue
« Reply #3 on: October 03, 2017, 01:33:20 AM »
Quote
Apparently, on linux, OPHD/NAS2D searches for config files and assets in ~/.lom, but the files aren't automatically copied there.

One workaround is to manually copy them. A better solution would be to either automatically copy the files or use the same layout as in windows with the config files and assets sitting alongside the rest of the installation.
Modify message

:o
I never would have thought to just copy the files there. Apparently I'm not so creative.

Though I would expect assets to be located alongside the binary. I'd also expect the saved game files to be located in a subfolder of the user dir. Makes sense to also put user adjustable config files there too.

I haven't taken a look at the config file. Actually, I can't seem to even find any config files. Is it all user adjustable settings, or does it contain game specific configuration set by the game designer? If it contains both, it should probably be split into two files.

Assuming the config file is all user settable, I suppose you could have a template master config file in the main game folder, which can be used as a base. From what I understand of physfs, if the user folder has a higher priority than the game binary folder, the master file would be used up until the point that a user specific file was created. That seems like very workable behaviour.


Physfs seems to support choosing and creating a folder for user specific saved game files:
PHYSFS_getPrefDir()
For Linux it returns something like "/home/bob/.local/share/My Program Name", while for Windows it returns something like  "/Users/bob/Library/Application Support/My Program Name".

According to the XDG Base Directory Specification, user specific data files would be stored in the locations $XDG_DATA_HOME ($HOME/.local/share), which physfs returns, while user specific config files should be stored in $XDG_CONFIG_HOME ($HOME/.config).

I'd say just dump both saved game files and user config in the folder returned by PHYSFS_getPrefDir.


Edit: Just noticed getPrefDir is for Physfs 3.0, while the system package for Ubuntu is only 2.0.3.  :-\
« Last Edit: October 03, 2017, 10:28:34 AM by Hooman »

Offline leeor_net

  • Administrator
  • Hero Member
  • *****
  • Posts: 2350
  • OPHD Lead Developer
    • LairWorks Entertainment
Re: Linux Data Folder Issue
« Reply #4 on: October 03, 2017, 06:43:16 PM »
I'm using 2.1.0 which has the function PHYSFS_getUserDir() but has been available since 1.0 so this is probably the function that should be called.

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4954
Re: Linux Data Folder Issue
« Reply #5 on: October 03, 2017, 08:26:38 PM »
Yeah, that would ease porting, since no source install for physfs would be needed. Though I should point out that getUserDir has been deprecated in the newest release.

Offline leeor_net

  • Administrator
  • Hero Member
  • *****
  • Posts: 2350
  • OPHD Lead Developer
    • LairWorks Entertainment
Re: Linux Data Folder Issue
« Reply #6 on: October 04, 2017, 05:07:51 PM »
We've seen that using the latest versions of libraries can be painful... as much as I'd love to port to PhysFS 3.0.0, after the fun with SDL 2.0.5 I'm game to stick with the most recent available version via get.

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4954
Re: Linux Data Folder Issue
« Reply #7 on: October 05, 2017, 01:40:06 AM »
I assume you meant apt get, in which case, thank you. :P

I think I know enough about the function now to take a stab at the changes myself.


Edit: Still had a few things to learn. Out of curiosity, I thought I'd see how hard a source install of PhysFS 3.0 would be. Turned out to be fairly easy.

Code: [Select]
wget https://icculus.org/physfs/downloads/physfs-3.0.0.tar.bz2
tar -jxvf physfs-3.0.0.tar.bz2
cd physfs-3.0.0/
cmake
make
sudo make install

It only takes a few seconds to run. Maybe build commands could be added to the make file. It would mean running the source install steps each time a TravisCI test was run. Though NAS2D has also been relatively stable after the initial Linux porting work.

The new functions are indeed very tempting to use. Plus the older functions have been deprecated. According to PHYSFS_getUserDir(), the change was made in version 2.1, earlier than I originally thought.
« Last Edit: October 05, 2017, 04:25:43 AM by Hooman »

Offline lhark

  • Newbie
  • *
  • Posts: 11
Re: Linux Data Folder Issue
« Reply #8 on: October 05, 2017, 09:24:38 AM »
On debian/ubuntu, you should replace `sudo make install` by `sudo checkinstall` so that the package is tracked by the packet manager and you don't end up with random files spread around your filesystem :)

Offline leeor_net

  • Administrator
  • Hero Member
  • *****
  • Posts: 2350
  • OPHD Lead Developer
    • LairWorks Entertainment
Re: Linux Data Folder Issue
« Reply #9 on: October 05, 2017, 05:14:53 PM »
I assume you meant apt get, in which case, thank you. :P

Yus... you see how much experience with Linux I have. I sort of know the commands but not quite... enough, at least, to google them.

And huh, I didn't realize it was so early in the game. But yeah, that's the big glaring thing in the Filesystem code that's been bugging me for awhile.

Offline Hooman

  • Administrator
  • Hero Member
  • *****
  • Posts: 4954
Re: Linux Data Folder Issue
« Reply #10 on: October 06, 2017, 03:51:08 AM »
On debian/ubuntu, you should replace `sudo make install` by `sudo checkinstall` so that the package is tracked by the packet manager and you don't end up with random files spread around your filesystem :)

Thank you for that! I'm wondering now if this can be used to speed up the TravisCI tests. A lot of the time is spent on the SDL2 compiling. Though the .deb packages would need to be hosted somewhere and downloaded by the test VM to install from. Not sure how appropriate it is to put them in the repo itself.



... enough, at least, to google them.

That's all that really matters  ;)

But yeah, that's the big glaring thing in the Filesystem code that's been bugging me for awhile.

I've been finding myself hesitant to make changes to your code. I think this will be a breaking change, in terms of source code interface compatibility. It's got me wondering about intent, and what your design goals are. It seems like PhysFS handles the default common case well. Could just delegate stuff to PhysFS without much change in semantics or complicated wrapping. Not sure if you're trying to build any restrictions or abstractions on top of it.

I think the PHYSFS_setSaneConfig does exactly what you want. It sets a read path where the executable is installed, allowing you to load game assets, and a read/write path in an appropriate subdirectory of the user's home directory, for things like saved games, or user specific config.

The one difference I think I see, is you seem to be adding subpaths to where the executable is installed, whereas the default is the parent folder containing the executable. I think perhaps the default config makes a lot of sense, and path to assets can be adjusted to include the subfolder.

Oh, and the setSaneConfig method can also search the read folder for archives with a given extension, and automatically add them. That can save you from manually specifying archives to load, and also eliminate error messages from adding archives you haven't created yet.

If a game doesn't want the default config, maybe the initialization of PHYSFS and the config of folders should be split into separate methods. Or maybe the default can run, but can be cleared or undone later. Or maybe you just want to disallow that, and say the default config should be good enough for any game likely to be created using NAS2D. Which might actually be very true. I'm thinking just stick with what setSaneConfig provides and not worry about it.


I was thinking the Filesystem::init could be something like this:
Code: [Select]
void Filesystem::init(const std::string& argv_0, const std::string& organizationName, const std::string& appName, const std::string& archiveExt)
{
if (FILESYSTEM_INITIALIZED) { throw filesystem_already_initialized(); }

std::cout << "Initializing Filesystem... ";

if (PHYSFS_init(argv_0.c_str()) == 0)
{
throw filesystem_backend_init_failure(PHYSFS_getLastError());
}

if (PHYSFS_setSaneConfig(organizationName, appName, archiveExt, false, false) == 0)
{
std::cout << std::endl << PHYSFS_getLastError() << std::endl;
}

FILESYSTEM_INITIALIZED = true;

std::cout << "done." << std::endl;
}

Was also thinking some of those messages might be more appropriate for cerr rather than cout. Not that it much matters for a graphical application.
« Last Edit: October 06, 2017, 03:59:52 AM by Hooman »