Adding Music Without A New Clm

[Sirbomber]

So, (if I can) how would I go about adding music for OP2 to use without making a new op2.clm?

Shortened: I want to use newOP2.wav in a mission and don't want to make a new op2.clm.

[zigzagjoe]

you could perhaps use some other thing to paly it back, eg not using op2 at all. would open it up to use mp3 and such as well. otherwise, no

[Mcshay]

Then how was it done in Renagades? All of the Renegades music seems to be not in a .clm.

[Hooman]

Oh, I sent Eddy some code that hooks into the OP2 sound system and fills the music buffer directly. It was actually quite involved. The stuff I handed over to Eddy was pretty raw but maybe he'll release something nice for everyone that people without his expertise could use.


Anyways, I have some code still around. I'm not sure how old it is. I know one of the earlier version I sent Eddy had some bugs in it. I think this is the patched version. Maybe get him to verify, but here's the basic stuff that gets things working.



MusicHook.cpp
-----------------

Code: [Select]

#include <memory.h>
#include "MusicHook.h"


// Addresses of Outpost2 code and data
DWORD outpost2ModuleBase = 0x00400000;	// Expected module base. Used for recalculating addresses
CRITICAL_SECTION *lpTimerCriticalSection = (CRITICAL_SECTION*)0x00565420;  // Critical section controlling access to the timer for the music buffer filling
DWORD hookAddress = 0x00450ED9;
DWORD hookReturn = 0x00450F28;
DWORD clmLoadAddress = 0x00450340;
DWORD managePlayListAddress = 0x00450DC0;
DWORD firstPlayAddress = 0x00450E63;
DWORD memAllocAddr = 0x004C0F40;
DWORD musicManagerAddr = 0x00565390;

const unsigned char hookedCodeBytes[] = {
	0xA1, 0x68, 0x54, 0x56, 0x00,  // MOV EAX, MusicManager.currentSongFileIndex
	0x8B, 0x0D, 0xAC, 0x53, 0x56, 0x00,	// MOV ECX, MusicManager.headerData*
	0xC1, 0xE0, 0x04,    	// SHL EAX, 4
	0x8B, 0x15, 0x6C, 0x54, 0x56, 0x00,	// MOV EDX, MusicManager.currentSongPosition
};

const unsigned char newCodeBytes[] = {
	0x8B, 0x4C, 0x24, 0x10,    // MOV ECX, [ESP+0x10] (AudioPtr1)
	0x68, 0x00, 0x80, 0x00, 0x00,  // PUSH 0x8000
	0x51,        // PUSH ECX (AudioPtr1)
	0x68,        // PUSH returnAddress
  0x00, 0x00, 0x00, 0x00,  	//  (returnAddress is written here)
	0xE9,        // JMP musicBufferFillingFunction
  0x00, 0x00, 0x00, 0x00,  	//  (musicBufferFillingFunction relative address is written here)
};

const unsigned char managePlayListBytes[] = {
	0xA1, 0x6C, 0x54, 0x56, 0x00  // MOV EAX, MusicManager.currentSongPosition
};

const unsigned char newPlayListSkip[] = {
	0xE9, 0xB9, 0x00, 0x00, 0x00,  // JMP FillMusicBuffer
};

const unsigned char firstPlayBytes[] = {
	0x8B, 0x14, 0x81,    	// MOV EDX, [ECX + EAX*4]
};

const unsigned char newFirstPlay[] = {
	0x33, 0xD2,      	// XOR EDX, EDX
	0x90        // NOP
};

const unsigned char clmLoadBytes[] = {
	0x81, 0xEC, 0x58, 0x01, 0x00, 0x00,	// SUB ESP, 0x158
};

const unsigned char newClmLoad[] = {
	0xB8, 0x01, 0x00, 0x00, 0x00,  // MOV EAX, 1
	0xC3,        // RETN
};


// **TODO** VERIFY THIS WORKS!
// **TODO** Find out if there is even much point? The hooked code contains memory addresses
//	that are being verified against anyways. :(
// Used to update all expected memory addresses of things in Outpost2.exe
void RebaseCode(DWORD newBaseAddress)
{
	// Calculate the offset between new and old addresses
	DWORD offset = newBaseAddress - outpost2ModuleBase;

	// Update current load address
	outpost2ModuleBase += offset;
	// Update the addresses
	*(char**)&lpTimerCriticalSection += offset;
	hookAddress += offset;
	hookReturn += offset;
	clmLoadAddress += offset;
	managePlayListAddress += offset;
	firstPlayAddress += offset;
	memAllocAddr += offset;
	musicManagerAddr += offset;
}


bool InitializeClmLoad()
{
	int headerDataAddr;

	// Allocate space for header data
	_asm
	{
  PUSH 0x4C
  CALL [memAllocAddr]
  MOV headerDataAddr, EAX
	}

	if (headerDataAddr == NULL)
  return false;

	// Fill in a fake CLM header
	memcpy((void*)headerDataAddr, "OP2 Clump File Version 1.0\0x1A\0x0\0\0\0\0", 32);
	WAVEFORMATEX *waveFormat = (WAVEFORMATEX*)(headerDataAddr + 0x32);
	waveFormat->wFormatTag = 1;
	waveFormat->nChannels = 1;
	waveFormat->nSamplesPerSec = 22050;
	waveFormat->nAvgBytesPerSec = 22050*2;
	waveFormat->nBlockAlign = 2;
	waveFormat->wBitsPerSample = 16;
	waveFormat->cbSize = 0;
	*(short*)(headerDataAddr+0x32) = 0;
	*(int*)(headerDataAddr+0x34) = 0x10000;
	*(int*)(headerDataAddr+0x38) = 0;  	// numPackedFiles
	// Fill in a fake index entry in the CLM header
	memset((void*)(headerDataAddr + 0x3C), 0, 8);
	*(int*)(headerDataAddr+0x44) = 0;
	*(int*)(headerDataAddr+0x48) = 0;


	// Fill in needed MusicManager fields
	*(int*)(musicManagerAddr + 0x1C) = headerDataAddr;	// char *headerData
	*(int*)(musicManagerAddr + 0x20) = -1;    // HANDLE hClmFile
	*(int*)(musicManagerAddr + 0x24) = 0x4C;  	// int totalFileHeaderSize

	// Fill in the song index table (point to only dummy index entry)
	int i;
	int *addr = (int*)(musicManagerAddr + 0x28);

	for (i = 0; i < 26; i++)
  addr[i] = 0;

	return true;
}

int InstallMusicHook(MusicBufferFillingFunc *musicPump)
{
	// Make sure the hooking location contains the correct code
	if (memcmp((void*)hookAddress, hookedCodeBytes, sizeof(hookedCodeBytes)) != 0)
  return 1;	// Failed. Unexpected instructions at hook point.
	// Make sure the clm loading location contains the correct code
	if (memcmp((void*)clmLoadAddress, clmLoadBytes, sizeof(clmLoadBytes)) != 0)
  return 2;	// Failed. Unexpected instructions at hook point.
	if (memcmp((void*)managePlayListAddress, managePlayListBytes, sizeof(managePlayListBytes)) != 0)
  return 3;	// Failed. Unexpected instructions at hook point.
	if (memcmp((void*)firstPlayAddress, firstPlayBytes, sizeof(firstPlayBytes)) != 0)
  return 4;	// Failed. Unexpected instructions at hook point.

	// Try to unprotect the code section
	DWORD oldAttributes;
	if (VirtualProtect((LPVOID)hookAddress, sizeof(hookedCodeBytes), PAGE_EXECUTE_READWRITE, &oldAttributes) == 0)
  return 5;  	// Could not unprotect pages. Abort
	if (VirtualProtect((LPVOID)clmLoadAddress, sizeof(clmLoadBytes), PAGE_EXECUTE_READWRITE, &oldAttributes) == 0)
  return 6;  	// Could not unprotect pages. Abort
	if (VirtualProtect((LPVOID)managePlayListAddress, sizeof(managePlayListBytes), PAGE_EXECUTE_READWRITE, &oldAttributes) == 0)
  return 7;  	// Could not unprotect pages. Abort
	if (VirtualProtect((LPVOID)firstPlayAddress, sizeof(firstPlayBytes), PAGE_EXECUTE_READWRITE, &oldAttributes) == 0)
  return 8;  	// Could not unprotect pages. Abort


	// Enter (timer) Critical Section
	// Note: The code we are overwriting is inside a critical section. By entering
	//	the critical section ourselves, we ensure the code is not executed while we
	//	are in the process of overwriting it.
	EnterCriticalSection(lpTimerCriticalSection);

	// Insert the hook
	// ***************
	memcpy((void*)hookAddress, newCodeBytes, sizeof(newCodeBytes));
	// Make sure the correct return address and function address are used
	*(DWORD*)&(((char*)hookAddress)[11]) = hookReturn;
	*(DWORD*)&(((char*)hookAddress)[16]) = (DWORD)musicPump - hookAddress - sizeof(newCodeBytes);
	// Patch up the CLM load
	memcpy((void*)clmLoadAddress, newClmLoad, sizeof(newClmLoad));
	memcpy((void*)managePlayListAddress, newPlayListSkip, sizeof(newPlayListSkip));
	memcpy((void*)firstPlayAddress, newFirstPlay, sizeof(newFirstPlay));

	// Initialized CLM data for safety (since load code is now skipped)
	InitializeClmLoad();

	// Leave (timer) Critical Section
	LeaveCriticalSection(lpTimerCriticalSection);


	// Return success
	return 0;
}

int UninstallMusicHook()
{
	// Check for an installed hook
	// Meh. Doesn't matter. Just overwriting with original code anyways

	// Enter (timer) Critical Section
	// Note: The code we are overwriting is inside a critical section. By entering
	//	the critical section ourselves, we ensure the code is not executed while we
	//	are in the process of overwriting it.
	EnterCriticalSection(lpTimerCriticalSection);

	// Remove the hook
	memcpy((void*)hookAddress, hookedCodeBytes, sizeof(hookedCodeBytes));
	memcpy((void*)clmLoadAddress, clmLoadBytes, sizeof(clmLoadBytes));
	memcpy((void*)managePlayListAddress, managePlayListBytes, sizeof(managePlayListBytes));
	memcpy((void*)firstPlayAddress, firstPlayBytes, sizeof(firstPlayBytes));

	// Leave (timer) Critical Section
	LeaveCriticalSection(lpTimerCriticalSection);

	// Return success
	return 0;
}

MusicHook.h
---------------

Code: [Select]

#include <dsound.h>


// Define a typedef for the callback function
typedef int __stdcall MusicBufferFillingFunc(char *musicBuffer, int bufferSize);

void RebaseCode(DWORD newBaseAddress);
int InstallMusicHook(MusicBufferFillingFunc *musicPump);
int UninstallMusicHook();

Of course you still need to know how to use those two files.
First of all, that's just used to install and uninstall the hook used to fill the music buffer. You still have to write a function to do that. Assuming your hook function is named FillMusicBuffer, here's how you can (un)install it.

Code: [Select]

BOOL APIENTRY DllMain(HANDLE hModule, DWORD  ul_reason_for_call, LPVOID lpReserved)
{
	if (ul_reason_for_call == DLL_PROCESS_ATTACH) 
	{
  DisableThreadLibraryCalls((HMODULE)hModule);
  InstallMusicHook(&FillMusicBuffer);
	}

	if (ul_reason_for_call == DLL_PROCESS_DETACH)
	{
  UninstallMusicHook();
	}

    return TRUE;
}

This is of course a modified version of the DllMain found in Main.cpp in the standard SDK release. Again, check with Eddy to see if what I had is really what worked in Renegades. I tend to make mistakes and I haven't looked at this stuff in ages so I really don't remember much about it anymore.



And again, this is only for installing and uninstalling the music hook. You still need to define the function that fills the music buffer.
 

[Hooman]

And if you want to test out the code, here's an overly simlified AUD decoder. It'll only play one song and then stop playing music. (But it does stop gracefully with silence and doesn't just stutter continuously at the end). Basically, it lacks any sort of playlist control. Well, here's the 6 files that do it.



AudPlayer.h
--------------

Code: [Select]

#include "MusicHook.h"


int __stdcall FillMusicBuffer(char *musicBuffer, int bufferSize);

AudPlayer.cpp
-----------------

Code: [Select]

#include "MusicHook.h"
#include "AudFileReader.h"
#include "AudDecoder.h"



// Note: This function fills the music buffer
//	with 0x8000 bytes worth of PCM data
// Note: Data format is 16 bit signed, mono, at 22050 Hz
// Return: Nonzero on success, zero on error. If an error occurs, 
//	the music buffer is cleared to zero.
int __stdcall FillMusicBuffer(char *musicBuffer, int bufferSize)
{
	DWORD numDecoded1;
	DWORD numAudBytesRead;
	static AudFileReader audReader;
	static AudDecoder audDecoder;
	static bool playNewAudFile = true;
	const DWORD audBufferSize = 0x2000;
	char audBuffer[audBufferSize];

	// Check if a new Aud file needs to be decoded
	if (playNewAudFile == true)
	{
  // **TODO** Make this work in the right way
  audReader.Open("ARAKATAK.AUD");
  audDecoder = AudDecoder();
  playNewAudFile = false;
	}

	// Read a chunk of Aud data
	numAudBytesRead = audReader.ReadAudData(audBuffer, audBufferSize);


	// Fill sound buffer
	// *****************
	numDecoded1 = audDecoder.Decode(musicBuffer, bufferSize, audBuffer, numAudBytesRead);
	memset(musicBuffer + numDecoded1, 0, bufferSize - numDecoded1);	// Zero fill if short on data

	// Return success
	return 1;
}

AudFileReader.h
--------------------

Code: [Select]

#define WIN32_LEAN_AND_MEAN
#include <windows.h>
#include <fstream.h>


class AudFileReader
{
	public:
  AudFileReader();
  ~AudFileReader();

  int Open(char *fileName);        // Opens file and reads in the header
  int ReadAudData(char *buffer, unsigned int bufferSize);	// Returns a chunk of AUD data
  // Wave format info
  int GetFrequency();  	
  int GetOutputSize();
  int GetCompressionType();
  int GetBitsPerSample();
  int GetNumChannels();
	private:
  // Private data types
  #pragma pack(push, 1)
  // AUD File header
  struct AudHeader
  {
  	short samplesPerSec;	// Frequency
  	int fileSize;  	// Size of file (without header)
  	int outputSize;  	// Size of output data
  	char flags;    // Bit 0 = Stereo, Bit 1 = 16 bit
  	char type;    // 1 = WW compressed, 99 = IMA ADPCM
  };
  // Chunk header
  struct ChunkHeader
  {
  	short size;
  	short outputSize;
  	int id;
  };
  #pragma pack(pop)

  enum AudFlags
  {
  	FlagStereo = 1,
  	Flag16Bit = 2,
  };

  // Private functions
  int ReadChunk(char *buffer, int bufferSize);

  // Class variables
  //ifstream audFile;
  HANDLE hFile;
  AudHeader audHeader;
  ChunkHeader currentChunkHeader;
  int currentChunkOffset;
};

AudFileReader.cpp
----------------------

Code: [Select]

#include "AudFileReader.h"


AudFileReader::AudFileReader()
{
	hFile = INVALID_HANDLE_VALUE;
	currentChunkOffset = 0;
	memset(&audHeader, 0, sizeof(audHeader));
	memset(&currentChunkHeader, 0, sizeof(currentChunkHeader));
}

AudFileReader::~AudFileReader()
{
	// Close the file if it is open
	if (hFile != INVALID_HANDLE_VALUE)
  CloseHandle(hFile);
}


// Opens an AUD file and reads in the file header
// Return: 0 on success, nonzero on failure.
int AudFileReader::Open(char *fileName)
{
	DWORD numBytesRead;

	// Try to open the file
	hFile = CreateFile(fileName, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL | FILE_FLAG_SEQUENTIAL_SCAN, NULL);
	// Check for errors
	if (hFile == INVALID_HANDLE_VALUE)
	{
  // Error opening the file
  return 1;
	}


	// Read the file header
	ReadFile(hFile, &audHeader, sizeof(audHeader), &numBytesRead, NULL);

	// Check for errors
	if (numBytesRead != sizeof(audHeader))
	{
  // Error reading AUD header
  CloseHandle(hFile);
  return 2;
	}

	// Return success
	return 0;
}

// Fills the passed in buffer with AUD data, up to the maximum size of the buffer.
// Return: The number of bytes actually written to the buffer
int AudFileReader::ReadAudData(char *buffer, unsigned int bufferSize)
{
	DWORD numBytesRead;
	DWORD numBytesWritten = 0;
	DWORD readSize;

	// Try to fill the buffer with AUD data
	while(bufferSize > numBytesWritten)
	{
  // Check if the current chunk has no more data left
  if (currentChunkHeader.size <= currentChunkOffset)
  {
  	// We need to read in a new chunk
  	ReadFile(hFile, &currentChunkHeader, sizeof(currentChunkHeader), &numBytesRead, NULL);

  	// Check for errors
  	if (numBytesRead != sizeof(currentChunkHeader))
  	{
    // Error reading new chunk header. Abort.
    return numBytesWritten;
  	}
  	// Reset current chunk offset
  	currentChunkOffset = 0;
  }

  // Determine how many bytes to read from the current chunk
  readSize = currentChunkHeader.size - currentChunkOffset;
  if (readSize > bufferSize - numBytesWritten)
  	readSize = bufferSize - numBytesWritten;

  // Read the chunk data
  ReadFile(hFile, &buffer[numBytesWritten], readSize, &numBytesRead, NULL);

  // Adjust the counters
  numBytesWritten += readSize;
  currentChunkOffset += readSize;

  // Check for errors
  if (numBytesRead != readSize)
  {
  	// Error reading chunk data. Abort.
  	return numBytesWritten;
  }
	}

	// Return the number of bytes written
	return numBytesWritten;
}



int AudFileReader::GetFrequency()
{
	return audHeader.samplesPerSec;
}

int AudFileReader::GetOutputSize()
{
	return audHeader.outputSize;
}

int AudFileReader::GetCompressionType()
{
	return audHeader.type;
}

int AudFileReader::GetBitsPerSample()
{
	if ((audHeader.flags & Flag16Bit) == Flag16Bit)
  return 16;
	else
  return 8;
}

int AudFileReader::GetNumChannels()
{
	if ((audHeader.flags & FlagStereo) == FlagStereo)
  return 2;
	else
  return 1;
}

AudDecoder.h
-----------------

Code: [Select]

class AudDecoder
{
	public:
  AudDecoder();

  int Decode(void *pcmBuffer, int pcmBufferSize, char *audBuffer, int audBufferSize);
	private:
  // Audio decoding state
  int index;
  int curSample;

  // Private static constant tables used during decompression
  static const int indexAdjust[];
  static const int stepTable[];
};

AudDecoder.cpp
-------------------

Code: [Select]

#include "AudDecoder.h"


// Define the static tables used during decompression
const int AudDecoder::indexAdjust[] = {-1,-1,-1,-1,2,4,6,8};
const int AudDecoder::stepTable[] = {
	7,     8,     9,     10,    11,    12,     13,    14,    16,
	17,    19,    21,    23,    25,    28,     31,    34,    37,
	41,    45,    50,    55,    60,    66,     73,    80,    88,
	97,    107,   118,   130,   143,   157,    173,   190,   209,
	230,   253,   279,   307,   337,   371,    408,   449,   494,
	544,   598,   658,   724,   796,   876,    963,   1060,  1166,
	1282,  1411,  1552,  1707,  1878,  2066,   2272,  2499,  2749,
	3024,  3327,  3660,  4026,  4428,  4871,   5358,  5894,  6484,
	7132,  7845,  8630,  9493,  10442, 11487,  12635, 13899, 15289,
	16818, 18500, 20350, 22385, 24623, 27086,  29794, 32767 
};


// Constructor
AudDecoder::AudDecoder()
{
	// Initialize the decoding state
	index = 0;
	curSample = 0;
}


// Decompresses the data from the audBuffer into the pcmBuffer
// Return value: The number of bytes written to the pcm buffer.
// Parameters:
//	pcmBuffer - a pointer to the buffer to fill with signed 16 bit PCM samples
//	pcmBufferSize - size of the pcmBuffer in bytes. (not the size in shorts)
//	audBuffer - a pointer to the input aud data (consisting of 4 bit samples)
//	audBufferSize - size of the audBuffer in bytes. (not the number of samples)
// Note: The aud data is in 4 bit samples, and the pcm data is 16 bit signed samples
// Note: Data is assumed to be mono
int AudDecoder::Decode(void *pcmBuffer, int pcmBufferSize, char *audBuffer, int audBufferSize)
{
	int numSamples;
	int i;
	int code;
	bool signBit;
	int delta;

	// Determine maximum amount of data that can be processed
	if (pcmBufferSize/2 >= audBufferSize*2)
  numSamples = audBufferSize*2;  	// Number of samples is implicity even
	else
  numSamples = (pcmBufferSize/2) & ~1;	// Round down to even number of samples


	// Decode the chunk of data
	for (i = 0; i < numSamples; i++)
	{
  // Get the current code (full byte)
  code = audBuffer[i/2];
  // Adjust for correct nibble
  if ((i & 1) == 1)
  	code >>= 4;

  // Get the sign of the nibble
  signBit = ((code & 8) == 8);
  // Truncate code to remaining bits
  code &= 7;

  // Calculate delta. Last part minimizes errors
  delta =(stepTable[index]*code)/4 + stepTable[index]/8;

  // Adjust for sign
  if (signBit)
  	delta = -delta;

  // Calculate the new sample
  curSample += delta;
  if (curSample < -32768) curSample = -32768;
  else if (curSample > 32767) curSample = 32767;

  // Write out the current sample
  ((short*)pcmBuffer)[i] = curSample;

  // Adjust the index
  index += indexAdjust[code];
  if (index < 0) index = 0;
  else if (index > 88) index = 88;
	}

	// Return the number of bytes written
	return numSamples*2;
}

Ohh, but now I've gone and spilled the beans!
I'm not sure if I even told Eddy what I was doing with my hooking code. But there it is. My custom built .Aud decoder. It was based off of some file I found on the internet about the music in Dune 2000. It used AD-PCM (Adaptive Pulse Code Modulation) to encode each 16 bit sample as 4 bits. Lossy compression of course. Anyways, I wanted to make a "sandbox" level using the Dune 2000 maps, and figured it wouldn't be complete without the music (and sand worms! ).

Anyways, I imported the Dune 2000 tile sets into Outpost2 format and a bunch of the multiplayer maps as well. I think there was around 25 something odd maps or so. I can't remember anymore. I think there were bugs in at least two maps that caused crashes. Anyways, I did that well over a year ago, and still haven't got around to making any levels, so I guess it's safe to say I'll never get around to it. So don't get your hopes up and expect this to ever be finished. The project has essentially been abandoned for a rather long time now. Oh, and I never added OP2 specific tiles to the Dune2000 tile set. Namely the tubes, walls, and bulldozed terrain, along with rubble (common and rare) and marks left on the ground from vehicle explosions. (Yes Haxtor, that's the secret project I tried to get you to help me with so very long ago).

Oh, and my work on importing the Dune 2000 tileset is what lead me to start on the project that became the map editor backend. I was mainly interested in the tile sets at the time, but the tile sets and the map format are so closely tied together in OP2 that I ended up doing so much work on the map files that I figured Hacker could make use of it for a map editor.

Of course adding new units into OP2 is damn near impossible, so adding sand worms just never came close. Although I did learn a fair bit about the OP2 unit system and I'd say it's somewhat possible to add new units in limited ways if you really wanted to work hard. I never quite learned everything I would have needed, but it was enough that I'd have some idea where to go.


Anyways, the music part worked, aside from the playlist. (And the base tileset was imported nicely). Enjoy.

 

[Hooman]

Oh wait, I was a bit wrong on my source. The aud format stuff I found was in reference to Command & Conquer and C&C Red Alert. But Dune 2000 and the C&C games were all made by Westwood. Here is where I found the info on decoding .aud files.

Aud page
I used the file Aud3.txt.

It was some good stuff. Well written.


 

[Eddy-B]

I'd have to check the files .. see if i've made modifications, but i think other then adding some easy-setup functions i didn't make much of a mod here; and used the code as supplied by Hooman

[BlackBox]

At some point I'll have to post my ModPlayer class as well that I gave Eddy; it allows you to play .mod / xm / it / s3m files by using bass.dll.