I am interested in using Python to generate new levels. Does pymclevel have that functionality? Any examples for world creation would be really really appreciated. Thanks!
As far as I am aware, it doesn't have anything specifically for world creation. It is a relatively simple process however; as long as you set up the folder and level.dat correctly, pymclevel can read it (and so can minecraft).
I wrote a script to generate flat worlds, with configurable numbers of hollow layers and a pre-set player inventory and a creative-in-a-box chest already placed. I've posted the source at http://github.com/xgkkp/mcmaketest if you want to see how I did it; it's mainly just creating the NBT structure. You don't even need to make the player information to have a world be loadable, but I wanted to have a convenient "ready to go" test world. It requires pymclevel and nbt.
One problem, that I haven't sorted out yet, is that I generate the player information from scratch, and it loads inside minecraft fine, but after saving and quitting, and reloading minecraft, it loses all player information. I don't understand this behaviour yet, as I thought I got all the required player tags in there, and it loads fine (I would have thought that it would write over the level.dat with a perfectly valid stucture).
I finished moving the mcmaketest code to use pymclevel instead of the other NBT, and I am not seeing this undesired behavior from saving, quitting and reloading. The only bit of mcmaketest that didn't transfer cleanly was the inventory hacking, and that's because the support just isn't there in pymclevel for it. I posted an issue on the github project with my patch, but it looks tres ugly there so just let me know if you want me to email it or what.
Now that I know how mcmaketest works, I'm very excited about using these concepts in my own code. Thank you!
Thanks a lot! Luckily github mails the issue plaintext so I just pulled it out from there, though it wasn't as easy as a direct push from a forked project (aww, it would have been fun to do that). Glad my code was of some use to you. TopoMC looks cool...
Next time I'll fork the project -- I was too chicken this time.
I'm making some progress porting TopoMC's output from JCraft back into Python. It's slower than a slow thing, but I think it's working. Have to let it go overnight at this rate to see for sure!
Okay, this rules. It is faster than a speeding bullet -- alas, it chews through an awful lot of memory, but I'm doing what I can about that. I am curious at how to make schematics. I have some structures I'd like to template out with schematics and finish once I'm in the game, but to do that I have to learn how to make them. Or should I use the editor? Help. :-)
Me again. Any chance you've put any thought into making pymclevel thread-safe? I've started using a module which makes numpy arrays multiprocessing-friendly and while there is a performance impact on the single-threaded version it's more than made up by being able to use all four cores. If I get reaaaaally motivated I might try to add that but I figured you would probably do a better job the first time. :-)
Hey Jack, seems I didn't reply to any of your posts in this thread. Let's see...
To make a schematic:
from pymclevel import MCSchematic
schem = MCSchematic(shape=(5, 3, 5) ) #shape is (x,y,z) but...
schem.Blocks[0,0,1] = 1 #indexes are [x,z,y]
#...set more blocks
schem.saveToFile(output.schematic)
I am not 100% certain that pymclevel is thread-safe, but I know that you can do a few things safely if you make sure each thread operates on a separate chunk, and also set loadedChunkLimit and decompressedChunkLimit to 0 to disable the autolimiters which I am pretty sure are not thread-safe. Python has a Global Interpreter Lock which helps you out a little by making sure only one thread can run Python code at once. Some library calls (numpy, opengl...) will release the lock and allow another thread to run and that's what you have to watch out for.
And yes, I've heard of the new format more than a few times. I'm not putting out any code for it until the game itself updates, since I won't have my hands on the actual save files until then.
The latest version of pymclevel seems to have 1.3 save file format support, but there are some problems when I try to use it.
It will create a new world directory, but not the region subdirectory, when I create a world.
When I create the directory, I get further along. When I go to get a chunk, I'll get a mclevel.chunkNotPresent exception. At that point, I try to create the chunk and get a ValueError because the chunk exists. Oops. :-)
I can make my code available to you if that'll help. Thanks for all your hard work!
Excellent, glad to hear it. Remember to exit the game or server completely before doing edits now, since the game will tend to cache parts of the files that it really shouldn't.
I'm using mathuins topomc which uses pymclevel to make the actual world in the end. The world i'm trying to make is decently large. I think it ends up being about 7000x7000 or so, I don't quite remember but here is the output I get.
C:\TopoMC>buildworld.py --region Badlands --world Worlds/Badlands
Creating world from region Badlands
Traceback (most recent call last):
File "C:\TopoMC\BuildWorld.py", line 119, in <module>
sys.exit(main(sys.argv))
File "C:\TopoMC\BuildWorld.py", line 88, in main
mcmap.initWorld(args.world, minX, minZ, maxX, maxZ, processes)
File "C:\TopoMC\mcmap.py", line 64, in initWorld
arrayData[arrayKey] = SharedMemArray(zeros((chunkWidth,chunkWidth,chunkHeight),dtype=uint8))
File "C:\TopoMC\multinumpy.py", line 108, in __init__
self.data = ndarray_to_shmem(arr)
File "C:\TopoMC\multinumpy.py", line 44, in ndarray_to_shmem
arr.size)
File "C:\Python26\lib\multiprocessing\__init__.py", line 241, in RawArray
return RawArray(typecode_or_type, size_or_initializer)
File "C:\Python26\lib\multiprocessing\sharedctypes.py", line 57, in RawArray
return _new_value(type_)
File "C:\Python26\lib\multiprocessing\sharedctypes.py", line 37, in _new_value
wrapper = heap.BufferWrapper(size)
File "C:\Python26\lib\multiprocessing\heap.py", line 190, in __init__
block = BufferWrapper._heap.malloc(size)
File "C:\Python26\lib\multiprocessing\heap.py", line 170, in malloc
(arena, start, stop) = self._malloc(size)
File "C:\Python26\lib\multiprocessing\heap.py", line 92, in _malloc
arena = Arena(length)
File "C:\Python26\lib\multiprocessing\heap.py", line 38, in __init__
self.buffer = mmap.mmap(-1, self.size, tagname=self.name)
WindowsError: [Error 8] Not enough storage is available to process this command
I think there's a filehandle leak in pymclevel. I'm porting TopoMC to generate chunk-level arrays instead of images so I can handle larger regions, and my first large test of roughly 123k files gave me this traceback:
Traceback (most recent call last):
File "./BuildWorld.py", line 47, in <module>
File "./BuildWorld.py", line 41, in main
File "/home/jmt/git/TopoMC/mcarray.py", line 127, in loadArrays
File "/home/jmt/git/TopoMC/mcarray.py", line 112, in loadArray
File "../pymclevel/mclevel.py", line 3795, in createChunk
File "../pymclevel/mclevel.py", line 1792, in __init__
File "../pymclevel/mclevel.py", line 1898, in create
File "../pymclevel/mclevel.py", line 1906, in save
File "../pymclevel/mclevel.py", line 2870, in _saveChunk
File "../pymclevel/mclevel.py", line 2310, in saveChunk
File "../pymclevel/mclevel.py", line 2366, in _saveChunk
File "../pymclevel/mclevel.py", line 2155, in file
File "../pymclevel/mclevel.py", line 2149, in <lambda>
IOError: [Errno 24] Too many open files: 'Worlds/Portland/region/r.9.-8.mcr'
If I create a chunk and write to it and am done with it, how do I make it save to the region file or at least close the open file descriptors? Thanks!
The RegionFile in pymclevel doesn't hold file descriptors open. I just did a test and created 62500 chunks in 64 region files. The number of filehandles used by the process didn't change throughout. What do you have 123k of, exactly?
I have a region of Portland, Oregon, that's 1164 pixels by 1069 pixels. I then blow it up by a factor of five which is a total of 31107900 square blocks and then I save it into 123343 pairs of 16x256x16 uint8 numpy arrays, one for blocks and one for data. Each pair of arrays is saved in a .npz file with a name like 222x-338.npz. I have a small routine which opens each npz with numpy.load(), reads in the arrays, then sets the load object to None. It then creates a chunk and populates the chunk with the contents of the arrays. It's possible that numpy.load() is leaving the files open but if I'm reading the source correctly they're not. Do you have any suggestions?
Had a map that caused Minecraft to keep saying it was relocating chunks (which it didn't.) It spat out gigabytes of exceptions. The only thing that let me save the map was pymclevel. I went from trying old backups and thinking I was going to have to tell someone his map was toast, to having a working map, in a matter of minutes. THANK YOU!!!
I'm hoping you can help out a programming newbie. I was looking at your Sample Usage comments and I was testing out the masks, replacing all trees with the Birch variety. My questions are
#0: I think this searches all blocks in that chunk and does a comparison to materials.X.ID, which results in a 3 dimensional array using numPy (I haven't used that yet, but type() returns <type 'numpy.ndarray'>) of True/False. The True/False values type() returns numpy.bool_, does that some how contain a reference to the block? Or does it later loop all X/Y/Z cords, and If treeblock[x][z][y] == True: change block ID?
#1: The '= 2' turns the selected chunk into birch trees. I don't understand where the number two comes in. If I change it to something else, lets say 10, it will turn all the trees into regular trees.
#2: How does [treeBlocks] end up changing all of the blocks? Could you point me to where in the code it does that?
I'm hoping you can help out a programming newbie. I was looking at your Sample Usage comments and I was testing out the masks, replacing all trees with the Birch variety. My questions are
#0: I think this searches all blocks in that chunk and does a comparison to materials.X.ID, which results in a 3 dimensional array using numPy (I haven't used that yet, but type() returns <type 'numpy.ndarray'>) of True/False. The True/False values type() returns numpy.bool_, does that some how contain a reference to the block? Or does it later loop all X/Y/Z cords, and If treeblock[x][z][y] == True: change block ID?
#1: The '= 2' turns the selected chunk into birch trees. I don't understand where the number two comes in. If I change it to something else, lets say 10, it will turn all the trees into regular trees.
#2: How does [treeBlocks] end up changing all of the blocks? Could you point me to where in the code it does that?
Thanks for any help.
#0: That's right. You're comparing an array to a number and returning a boolean array with one element for each element in the original array. It's the same shape as the original array which means you can use the boolarray in indexing operations to access only the parts of the original array where the boolarray is true. It doesn't contain a reference to the original array or its elements, but what's important is being the same shape (x,y,z dimensions) so that numpy can correlate the elements in one array with those in the other array.
#2: [treeBlocks] is the numpy array indexing operator. In this instance, you're telling it to choose elements of the array based on the true/false values in treeBlocks. It will set only the elements of chunk.Data where the corresponding element of treeBlocks is true. (Specifically, it's the set-array-item operator, since you're typing chunk.Data[treeBlocks] = <something> in the same statement. If you were to just write chunk.Data[treeBlocks], the result of that would be a 1-D array of the elements of chunk.Data where treeBlocks is true. This is a subtle difference.)
#1: It could be better written as chunk.Data[treeBlocks] = world_0.materials.BirchWood.blockData. The Data array has "extra data" for each block in the Blocks array to define the wood species, wool color, or other things depending on the block type.
Thanks for the clarification, I do have one more question. Let's say some one, for what ever reason, wanted to turn all the trees in the world into lava, what would he set chunk.Data[treeBlocks] to?
Jack.
I finished moving the mcmaketest code to use pymclevel instead of the other NBT, and I am not seeing this undesired behavior from saving, quitting and reloading. The only bit of mcmaketest that didn't transfer cleanly was the inventory hacking, and that's because the support just isn't there in pymclevel for it. I posted an issue on the github project with my patch, but it looks tres ugly there so just let me know if you want me to email it or what.
Now that I know how mcmaketest works, I'm very excited about using these concepts in my own code. Thank you!
Jack.
Next time I'll fork the project -- I was too chicken this time.
I'm making some progress porting TopoMC's output from JCraft back into Python. It's slower than a slow thing, but I think it's working. Have to let it go overnight at this rate to see for sure!
Jack.
Jack.
(very very pleased, seriously.)
Jack.
Looks like there's a new save file format coming this way, didn't know if you knew.
Jack.
To make a schematic:
I am not 100% certain that pymclevel is thread-safe, but I know that you can do a few things safely if you make sure each thread operates on a separate chunk, and also set loadedChunkLimit and decompressedChunkLimit to 0 to disable the autolimiters which I am pretty sure are not thread-safe. Python has a Global Interpreter Lock which helps you out a little by making sure only one thread can run Python code at once. Some library calls (numpy, opengl...) will release the lock and allow another thread to run and that's what you have to watch out for.
And yes, I've heard of the new format more than a few times. I'm not putting out any code for it until the game itself updates, since I won't have my hands on the actual save files until then.
"We will absolutely not keep in mind what external mapeditors will have to do to read data from the disk, that makes no sense whatsoever." - Grum
It will create a new world directory, but not the region subdirectory, when I create a world.
When I create the directory, I get further along. When I go to get a chunk, I'll get a mclevel.chunkNotPresent exception. At that point, I try to create the chunk and get a ValueError because the chunk exists. Oops. :-)
I can make my code available to you if that'll help. Thanks for all your hard work!
Jack.
Jack.
Excellent, glad to hear it. Remember to exit the game or server completely before doing edits now, since the game will tend to cache parts of the files that it really shouldn't.
"We will absolutely not keep in mind what external mapeditors will have to do to read data from the disk, that makes no sense whatsoever." - Grum
If I create a chunk and write to it and am done with it, how do I make it save to the region file or at least close the open file descriptors? Thanks!
"We will absolutely not keep in mind what external mapeditors will have to do to read data from the disk, that makes no sense whatsoever." - Grum
Jack.
ETA: corrected array dimensions
"We will absolutely not keep in mind what external mapeditors will have to do to read data from the disk, that makes no sense whatsoever." - Grum
... and that proves that it's numpy being bad not your code. Thank you. :-)
Jack.
#0: I think this searches all blocks in that chunk and does a comparison to materials.X.ID, which results in a 3 dimensional array using numPy (I haven't used that yet, but type() returns <type 'numpy.ndarray'>) of True/False. The True/False values type() returns numpy.bool_, does that some how contain a reference to the block? Or does it later loop all X/Y/Z cords, and If treeblock[x][z][y] == True: change block ID?
#1: The '= 2' turns the selected chunk into birch trees. I don't understand where the number two comes in. If I change it to something else, lets say 10, it will turn all the trees into regular trees.
#2: How does [treeBlocks] end up changing all of the blocks? Could you point me to where in the code it does that?
Thanks for any help.
#0: That's right. You're comparing an array to a number and returning a boolean array with one element for each element in the original array. It's the same shape as the original array which means you can use the boolarray in indexing operations to access only the parts of the original array where the boolarray is true. It doesn't contain a reference to the original array or its elements, but what's important is being the same shape (x,y,z dimensions) so that numpy can correlate the elements in one array with those in the other array.
#2: [treeBlocks] is the numpy array indexing operator. In this instance, you're telling it to choose elements of the array based on the true/false values in treeBlocks. It will set only the elements of chunk.Data where the corresponding element of treeBlocks is true. (Specifically, it's the set-array-item operator, since you're typing chunk.Data[treeBlocks] = <something> in the same statement. If you were to just write chunk.Data[treeBlocks], the result of that would be a 1-D array of the elements of chunk.Data where treeBlocks is true. This is a subtle difference.)
#1: It could be better written as chunk.Data[treeBlocks] = world_0.materials.BirchWood.blockData. The Data array has "extra data" for each block in the Blocks array to define the wood species, wool color, or other things depending on the block type.
"We will absolutely not keep in mind what external mapeditors will have to do to read data from the disk, that makes no sense whatsoever." - Grum