Latest News Article

[Pjstaab]

I'm using mathuins topomc which uses pymclevel to make the actual world in the end.  The world i'm trying to make is decently large.  I think it ends up being about 7000x7000 or so, I don't quite remember but here is the output I get.

C:\TopoMC>buildworld.py --region Badlands --world Worlds/Badlands
Creating world from region Badlands
Traceback (most recent call last):
  File "C:\TopoMC\BuildWorld.py", line 119, in 
	sys.exit(main(sys.argv))
  File "C:\TopoMC\BuildWorld.py", line 88, in main
	mcmap.initWorld(args.world, minX, minZ, maxX, maxZ, processes)
  File "C:\TopoMC\mcmap.py", line 64, in initWorld
	arrayData[arrayKey] = SharedMemArray(zeros((chunkWidth,chunkWidth,chunkHeight),dtype=uint8))
  File "C:\TopoMC\multinumpy.py", line 108, in __init__
	self.data = ndarray_to_shmem(arr)
  File "C:\TopoMC\multinumpy.py", line 44, in ndarray_to_shmem
	arr.size)
  File "C:\Python26\lib\multiprocessing\__init__.py", line 241, in RawArray
	return RawArray(typecode_or_type, size_or_initializer)
  File "C:\Python26\lib\multiprocessing\sharedctypes.py", line 57, in RawArray
	return _new_value(type_)
  File "C:\Python26\lib\multiprocessing\sharedctypes.py", line 37, in _new_value
	wrapper = heap.BufferWrapper(size)
  File "C:\Python26\lib\multiprocessing\heap.py", line 190, in __init__
	block = BufferWrapper._heap.malloc(size)
  File "C:\Python26\lib\multiprocessing\heap.py", line 170, in malloc
	(arena, start, stop) = self._malloc(size)
  File "C:\Python26\lib\multiprocessing\heap.py", line 92, in _malloc
	arena = Arena(length)
  File "C:\Python26\lib\multiprocessing\heap.py", line 38, in __init__
	self.buffer = mmap.mmap(-1, self.size, tagname=self.name)
WindowsError: [Error 8] Not enough storage is available to process this command

[mathuin]

I think there's a filehandle leak in pymclevel.  I'm porting TopoMC to generate chunk-level arrays instead of images so I can handle larger regions, and my first large test of roughly 123k files gave me this traceback:

Traceback (most recent call last):
  File "./BuildWorld.py", line 47, in 
  File "./BuildWorld.py", line 41, in main
  File "/home/jmt/git/TopoMC/mcarray.py", line 127, in loadArrays
  File "/home/jmt/git/TopoMC/mcarray.py", line 112, in loadArray
  File "../pymclevel/mclevel.py", line 3795, in createChunk
  File "../pymclevel/mclevel.py", line 1792, in __init__
  File "../pymclevel/mclevel.py", line 1898, in create
  File "../pymclevel/mclevel.py", line 1906, in save
  File "../pymclevel/mclevel.py", line 2870, in _saveChunk
  File "../pymclevel/mclevel.py", line 2310, in saveChunk
  File "../pymclevel/mclevel.py", line 2366, in _saveChunk
  File "../pymclevel/mclevel.py", line 2155, in file
  File "../pymclevel/mclevel.py", line 2149, in 
IOError: [Errno 24] Too many open files: 'Worlds/Portland/region/r.9.-8.mcr'

If I create a chunk and write to it and am done with it, how do I make it save to the region file or at least close the open file descriptors?  Thanks!

[codewarrior]

The RegionFile in pymclevel doesn't hold file descriptors open. I just did a test and created 62500 chunks in 64 region files. The number of filehandles used by the process didn't change throughout. What do you have 123k of, exactly?

[mathuin]

I have a region of Portland, Oregon, that's 1164 pixels by 1069 pixels.  I then blow it up by a factor of five which is a total of 31107900 square blocks and then I save it into 123343 pairs of 16x256x16 uint8 numpy arrays, one for blocks and one for data.  Each pair of arrays is saved in a .npz file with a name like 222x-338.npz.  I have a small routine which opens each npz with numpy.load(), reads in the arrays, then sets the load object to None.  It then creates a chunk and populates the chunk with the contents of the arrays.  It's possible that numpy.load() is leaving the files open but if I'm reading the source correctly they're not.  Do you have any suggestions?

Jack.

ETA: corrected array dimensions

[codewarrior]

Catch the IOError somewhere and sleep on it.  Then, use lsof on the sleeping python process to see which files are open.

[mathuin]

codewarrior said:

Catch the IOError somewhere and sleep on it.  Then, use lsof on the sleeping python process to see which files are open.

... and that proves that it's numpy being bad not your code.  Thank you. :-)

Jack.

[Huns]

Had a map that caused Minecraft to keep saying it was relocating chunks (which it didn't.) It spat out gigabytes of exceptions. The only thing that let me save the map was pymclevel. I went from trying old backups and thinking I was going to have to tell someone his map was toast, to having a working map, in a matter of minutes. THANK YOU!!!

[akaGrim]

I'm hoping you can help out a programming newbie. I was looking at your Sample Usage comments and I was testing out the masks, replacing all trees with the Birch variety. My questions are

treeBlocks = (mychunk.Blocks==world_0.materials.Leaves.ID) #0
treeBlocks |= (mychunk.Blocks==world_0.materials.Wood.ID)  
chunk.Data[treeBlocks] = 2 #1							  #2 

#0: I think this searches all blocks in that chunk and does a comparison to materials.X.ID, which results in a 3 dimensional array using numPy (I haven't used that yet, but type() returns <type 'numpy.ndarray'>) of True/False. The True/False values type() returns numpy.bool_, does that some how contain a reference to the block? Or does it later loop all X/Y/Z cords, and If treeblock[x][z][y] == True: change block ID?

#1: The '= 2' turns the selected chunk into birch trees. I don't understand where the number two comes in. If I change it to something else, lets say 10, it will turn all the trees into regular trees.

#2: How does [treeBlocks] end up changing all of the blocks? Could you point me to where in the code it does that?

Thanks for any help.

[codewarrior]

akaGrim, on 23 July 2011 - 12:37 AM, said:

I'm hoping you can help out a programming newbie. I was looking at your Sample Usage comments and I was testing out the masks, replacing all trees with the Birch variety. My questions are

treeBlocks = (mychunk.Blocks==world_0.materials.Leaves.ID) #0
treeBlocks |= (mychunk.Blocks==world_0.materials.Wood.ID)  
chunk.Data[treeBlocks] = 2 #1							  #2 

#0: I think this searches all blocks in that chunk and does a comparison to materials.X.ID, which results in a 3 dimensional array using numPy (I haven't used that yet, but type() returns <type 'numpy.ndarray'>) of True/False. The True/False values type() returns numpy.bool_, does that some how contain a reference to the block? Or does it later loop all X/Y/Z cords, and If treeblock[x][z][y] == True: change block ID?

#1: The '= 2' turns the selected chunk into birch trees. I don't understand where the number two comes in. If I change it to something else, lets say 10, it will turn all the trees into regular trees.

#2: How does [treeBlocks] end up changing all of the blocks? Could you point me to where in the code it does that?

Thanks for any help.

#0:  That's right. You're comparing an array to a number and returning a boolean array with one element for each element in the original array. It's the same shape as the original array which means you can use the boolarray in indexing operations to access only the parts of the original array where the boolarray is true. It doesn't contain a reference to the original array or its elements, but what's important is being the same shape (x,y,z dimensions) so that numpy can correlate the elements in one array with those in the other array.

#2: [treeBlocks] is the numpy array indexing operator. In this instance, you're telling it to choose elements of the array based on the true/false values in treeBlocks.  It will set only the elements of chunk.Data where the corresponding element of treeBlocks is true.  (Specifically, it's the set-array-item operator, since you're typing chunk.Data[treeBlocks] = <something> in the same statement. If you were to just write chunk.Data[treeBlocks], the result of that would be a 1-D array of the elements of chunk.Data where treeBlocks is true. This is a subtle difference.)

#1: It could be better written as chunk.Data[treeBlocks] = world_0.materials.BirchWood.blockData.  The Data array has "extra data" for each block in the Blocks array to define the wood species, wool color, or other things depending on the block type.

[akaGrim]

Thanks for the clarification, I do have one more question. Let's say some one, for what ever reason, wanted to turn all the trees in the world into lava, what would he set chunk.Data[treeBlocks] to?

[codewarrior]

akaGrim, on 23 July 2011 - 06:07 PM, said:

Thanks for the clarification, I do have one more question. Let's say some one, for what ever reason, wanted to turn all the trees in the world into lava, what would he set chunk.Data[treeBlocks] to?

You have to set chunk.Blocks[treeBlocks] to change the block type to Lava (Still), and then set chunk.Data to change the lava's level to 0, which means a full source block. If you want the game to process the lava and make it flow the next time you play, use LavaActive instead.

chunk.Blocks[treeBlocks] = alphaMaterials.LavaStill.ID
chunk.Data[treeBlocks] = 0

[akaGrim]

codewarrior, on 24 July 2011 - 02:54 AM, said:

chunk.Blocks[treeBlocks] = alphaMaterials.LavaStill.ID
chunk.Data[treeBlocks] = 0

Some reason I'm getting an error with your code;
NameError: name 'alphaMaterials' is not defined

[xolotl]

akaGrim, on 25 July 2011 - 03:27 AM, said:

Some reason I'm getting an error with your code;
NameError: name 'alphaMaterials' is not defined

If you've only done an "import mclevel" up at the top, you wouldn't have that name defined.  Try adding "from materials import alphaMaterials" (or just "import materials" and reference it by the "full" name).

[akaGrim]

xolotl, on 25 July 2011 - 04:31 AM, said:

If you've only done an "import mclevel" up at the top, you wouldn't have that name defined.  Try adding "from materials import alphaMaterials" (or just "import materials" and reference it by the "full" name).

I figured it out myself this morning. mclevel does a 'from materials import alphaMaterials' so I just used mclevel.alphaMaterials and then it worked.

[frymaster]

Just occurred to me that I should probably say thanks for this.  It came in handy when lapis lazuli was introduced - I was able to fairly easily write something to turn 10% of all the redstone in the world into lapis lazuli:

http://pastebin.127001.org/1741

Here's a screenshot of the test run I did where I turned 10% of all the air blocks into lapis (and stopped the script before it had processed all the chunks, obviously)
http://img.127001.or...-lot-of-ore.png

[liamf91]

can any one give me a hand in sorting this and another tool to work?

i posted a thread

http://www.minecraft..._1#entry7662782

[mathuin]

How hard would it be to use pymclevel to merge two levels into one?  Assume the two worlds have no intersection and that they are adjacent, if that helps!  Thanks!

Jack.

[codewarrior]

Um...really simple? The only thing left is to calculate destinationPoint so I'll leave it up to you to position the incoming world. If the two worlds really are not overlapping with regards to the global coordinate system, you can use 'world2.bounds.origin' for destinationPoint.

world1.copyBlocksFrom(world2, world2.bounds, destinationPoint)

[Eggplant!]

Congrats on the MCServerChunkGenerator stuff. Seriously. It's incredible.

Quick question: Is there a way to specify which generator gets used? (Skylands vs Nether vs Normal)

[codewarrior]

Eggplant!, on 01 September 2011 - 03:01 PM, said:

Congrats on the MCServerChunkGenerator stuff. Seriously. It's incredible.

Quick question: Is there a way to specify which generator gets used? (Skylands vs Nether vs Normal)

I don't know about Skylands because I haven't seen a server mod for it. However, the generator is aware of dimension numbers and will try to generate chunks from the matching dimension.

There's no way to explicitly ask for Nether terrain, but it will be generated whenever you ask it to generate chunks into a Nether level. If you want to have Nether terrain on Earth, you'll have to explicitly use copyBlocksFrom to put it there.

The above is notoriously incompatible with Bukkit since Bukkit changes the directory layout of multidimensional worlds.