To get you started:
The original tiles are 32x32 pixels with 8bit color (palettized). Several tiles may be stored in one bmp-file as one column of tiles (and nothing else). Original op2 uses some non-standard-bmp, but regular bmp should also work as long as it obeys the restrictions above.
In order to use those tiles, you may have to put them into your op2 directory or even into the vol-files (the mapper can pack/unpack vol-files; but you better do it with an extra op2 install to keep multiplayer working).
You should be able to use your own tiles just with the mapper. You can add new tilesets to any existing map, but most maps using the original tilesets seem to use almost the maximum number of tiles, so it might be a good idea to remove unused original tilesets or even start with none of them (when creating a new map).
After adding tilesets to a map, the map needs some info on special tiles (walls, tubes, lava, etc.). Its called something like "Terrain" in the mapper and accassible over a button in the map window.
You'll have to figure out the details on your own, as I never really created my own tileset.