16 Bit Output

[manticorp]

This is an amazing tool - the data is available in 16 bit format, but the restrictions of web canvases mean in grayscale the data is limited to 8bit, which is a shame, because for use in many game heightmap imports, 8 bit data is not granular enough and results in large steps.

Is it possible to toggle 16 bit output, in the form of the straight terrarium data output by the tangram api?

I think this would be possible by creating a shader that doesn't perform the conversion from the terrarium style data to grayscale, which necessarily clamps to 8bit when writing to the canvas, and being able to toggle that on for rendering.

A separate tool could then be created to convert that data into a 16bit grayscale PNG file.

[drakgoku]

Did you find any way?

[manticorp]

Did you find any way?

Yes - turns out the data written to the web canvas is really noisy, probably due to browser resampling and the data not being 1:1 for the display. Because I was using this to generate 3D terrain data, this was no good.

So in the end I wrote a python script that outputs the tiles directly and stitches them together. Also using python and numpy I translate the 8 bit colour PNGs to 16 bit grayscale (for unreal engine import).

[llleeexxxoooppp]

Did you find any way?

Yes - turns out the data written to the web canvas is really noisy, probably due to browser resampling and the data not being 1:1 for the display. Because I was using this to generate 3D terrain data, this was no good.

So in the end I wrote a python script that outputs the tiles directly and stitches them together. Also using python and numpy I translate the 8 bit colour PNGs to 16 bit grayscale (for unreal engine import).

If it's not secret.. how it should work, that python script? Please, if it possible, share answer , or probably has alternative method to access 16bit data? (i not strong in python, but me too saw "stairs" on mountains in blender, and that makes me sad how close good maps of whole planet but I don't have the required skills to access it). I will be grateful.

[manticorp]

Did you find any way?

Yes - turns out the data written to the web canvas is really noisy, probably due to browser resampling and the data not being 1:1 for the display. Because I was using this to generate 3D terrain data, this was no good.
So in the end I wrote a python script that outputs the tiles directly and stitches them together. Also using python and numpy I translate the 8 bit colour PNGs to 16 bit grayscale (for unreal engine import).

If it's not secret.. how it should work, that python script? Please, if it possible, share answer , or probably has alternative method to access 16bit data? (i not strong in python, but me too saw "stairs" on mountains in blender, and that makes me sad how close good maps of whole planet but I don't have the required skills to access it). I will be grateful.

Because I only had to generate a single map (I wasn't planning on doing this multiple times) my code for this is very hacky.

It actually is in three stages:

• A PHP script that downloads the tiles for a given area
• A Python script that stitches them together
• Another Python script that converts them to a 16 bit grayscale PNG

I have cleaned it up a bit and put it in a gist for convenience, with examples:

https://gist.github.com/manticorp/3759cdbb3488c529c7d1f0c9b3c74ccf

[llleeexxxoooppp]

Did you find any way?

Yes - turns out the data written to the web canvas is really noisy, probably due to browser resampling and the data not being 1:1 for the display. Because I was using this to generate 3D terrain data, this was no good.
So in the end I wrote a python script that outputs the tiles directly and stitches them together. Also using python and numpy I translate the 8 bit colour PNGs to 16 bit grayscale (for unreal engine import).

If it's not secret.. how it should work, that python script? Please, if it possible, share answer , or probably has alternative method to access 16bit data? (i not strong in python, but me too saw "stairs" on mountains in blender, and that makes me sad how close good maps of whole planet but I don't have the required skills to access it). I will be grateful.

Because I only had to generate a single map (I wasn't planning on doing this multiple times) my code for this is very hacky.

It actually is in three stages:

* A PHP script that downloads the tiles for a given area

* A Python script that stitches them together

* Another Python script that converts them to a 16 bit grayscale PNG

I have cleaned it up a bit and put it in a gist for convenience, with examples:

https://gist.github.com/manticorp/3759cdbb3488c529c7d1f0c9b3c74ccf

Well, after all the preparations, I was finally able to run the php script, but it doesn't work very well. Latitude and longitude can have negative values, but the script does not want to accept them, reporting invalid syntax.
> php renderLocation.php -a YOUR_API_KEY -l 36.34801 -k -112.45470 -x 8 -y 8 -z 13 -e
> ERROR: Unable to parse option -112.45470: Invalid syntax
I also tried to enter zero before minus (0-122.45470), and then the script parsed, but the final image was obviously broken. This means that it is possible to get an image only from the northeastern part of the planet, where the longitude and latitude values are positive, or I missed something and need to enter coordinates in a different way.
I was trying to figure out the code inside the php script, and as far as I understood, I need to rewrite how the coordinates are calculated.
But, of course, thank you so much for sharing and preparing the resources and wrote clearly read.me

[manticorp]

Haha that's quite possible as I happened to only need it for areas in northeastern hemispheres! I will have a look later as it sounds like a relatively simple fix 😊

[manticorp]

Did you find any way?

Yes - turns out the data written to the web canvas is really noisy, probably due to browser resampling and the data not being 1:1 for the display. Because I was using this to generate 3D terrain data, this was no good.
So in the end I wrote a python script that outputs the tiles directly and stitches them together. Also using python and numpy I translate the 8 bit colour PNGs to 16 bit grayscale (for unreal engine import).

If it's not secret.. how it should work, that python script? Please, if it possible, share answer , or probably has alternative method to access 16bit data? (i not strong in python, but me too saw "stairs" on mountains in blender, and that makes me sad how close good maps of whole planet but I don't have the required skills to access it). I will be grateful.

Because I only had to generate a single map (I wasn't planning on doing this multiple times) my code for this is very hacky.
It actually is in three stages:

* A PHP script that downloads the tiles for a given area

* A Python script that stitches them together

* Another Python script that converts them to a 16 bit grayscale PNG

I have cleaned it up a bit and put it in a gist for convenience, with examples:
https://gist.github.com/manticorp/3759cdbb3488c529c7d1f0c9b3c74ccf

Well, after all the preparations, I was finally able to run the php script, but it doesn't work very well. Latitude and longitude can have negative values, but the script does not want to accept them, reporting invalid syntax. > php renderLocation.php -a YOUR_API_KEY -l 36.34801 -k -112.45470 -x 8 -y 8 -z 13 -e > ERROR: Unable to parse option -112.45470: Invalid syntax I also tried to enter zero before minus (0-122.45470), and then the script parsed, but the final image was obviously broken. This means that it is possible to get an image only from the northeastern part of the planet, where the longitude and latitude values are positive, or I missed something and need to enter coordinates in a different way. I was trying to figure out the code inside the php script, and as far as I understood, I need to rewrite how the coordinates are calculated. But, of course, thank you so much for sharing and preparing the resources and wrote clearly read.me

So, I've been having a look, it was a bug in the CLI command library I was using.

I've updated and created a proper repo here:

https://github.com/manticorp/Nextzen16bitGrayscaleHeightmap

I've added some convenience stuff as well (config files, location storage, etc).

It should all work now! Any more issues please post them in the issues on the new repo.

I've even managed to export that map of the colorado river :)

[llleeexxxoooppp]

It's just wonderful, I don't know what to write, man, you're just cool.
With this tool, it will be possible to do so many things - render the entire planet or feed the neural network with a lot of highmaps to generate more maps, and more and more... ahaha (but, this is so.. imaginations).
I hope the original servers storing the data will still work for a long time.
I didn't think that asking for an answer for the first time would bring this thing to... well, I don't know, but you understand ahah. Initially, I wanted to find a procedural method for creating a natural landscape, an algorithm or a ready-made solution, but I did not find any other than descriptions in the article form. Ready-made maps are a large database of examples to learn how to imitate them. It is not difficult to make a simple deformation on a plane that will look like mountains, but canyons, sharp slopes, mountain and river erosion are more difficult. To do this, need to create a lot of masks for each factor separately, but without natural examples, it is not so easy to do this with the task of obtaining a realistic result. With the help of your tool, it will be much easier to get the most diverse types of landscape "here and now". Thank you for what time)

[manticorp]

It's just wonderful, I don't know what to write, man, you're just cool. With this tool, it will be possible to do so many things - render the entire planet or feed the neural network with a lot of highmaps to generate more maps, and more and more... ahaha (but, this is so.. imaginations). I hope the original servers storing the data will still work for a long time. I didn't think that asking for an answer for the first time would bring this thing to... well, I don't know, but you understand ahah. Initially, I wanted to find a procedural method for creating a natural landscape, an algorithm or a ready-made solution, but I did not find any other than descriptions in the article form. Ready-made maps are a large database of examples to learn how to imitate them. It is not difficult to make a simple deformation on a plane that will look like mountains, but canyons, sharp slopes, mountain and river erosion are more difficult. To do this, need to create a lot of masks for each factor separately, but without natural examples, it is not so easy to do this with the task of obtaining a realistic result. With the help of your tool, it will be much easier to get the most diverse types of landscape "here and now". Thank you for what time)

Amazing! I'm so glad that it worked for you, and you're pleased with the result. Thank you so much for taking the time to say so.

I have noticed a lot of people using the Tangram heightmapper website for unreal engine, and notice that they all suffer from the same low resolution and banding/jaggies. Perhaps I will release this as a standalone software, or perhaps a website, one day.

[drakgoku]

have taken the trouble to compare 3 images in UE5:

1 - An image of tangrams downloaded to 8 bits and converted to 16 bits (here you will lose information and it will look bad)
https://i.imgur.com/VrTbAKQ.png
2 - The river of this publication. It looks good.
https://i.imgur.com/XQ8MuG3.png
3 - The example map of your tool.Nextzen16bitGrayscaleHeightmap. It looks good.
https://i.imgur.com/L575rhh.png

The resulting effect is not so bad either. Let's say it's 90% good. They look pretty good with that script.

Why do you have the alpha channel disabled?

Can you apply this tool or that extra plugin when on the tangrams page so that when the user downloads he does it in 16 bits instead of 8 bits?

[manticorp]

Why do you have the alpha channel disabled?

I was using these images to generate heightmaps for the landscape feature in UE5.

Unreal Engine expects the data to be in 16 bit grayscale PNG - it does not support alpha channel for landscape heightmaps (and how would that work? Semitransparent landscape?)

1 - An image of tangrams downloaded to 8 bits and converted to 16 bits (here you will lose information and it will look bad)
https://i.imgur.com/VrTbAKQ.png

2^8 (i.e. regular RGB grayscale) only gives 256 height levels, and so at a resolution of 1m you only have 256m max elevation, or 10m resolution for 2560m elevation.

2^16 gives 65536 height levels, so for example at a resolution of 1m you could have a 65536m high mountain. I believe the mapzen terrarium data is at 10m with interpolation, so 2^16 (16 bit grayscale PNG) is more than enough.

You could theoretically have a 24 bit PNG for even greater resolution, as the Mapzen data has fractional metre info.

More information from Mapzen here

If one could write a terrarium reader for UE5 this would solve all these problems outright, though I'm not sure on the resolution used by UE5 internally - I expect it's only 16 bit anyway.

Can you apply this tool or that extra plugin when on the tangrams page so that when the user downloads he does it in 16 bits instead of 8 bits?

The way the map viewer works introduces 2 problems that I'm aware of:

1. The data is converted to grayscale in a way that does not support > 8bit encoding.
2. The way the tangram viewer is implemented on this tool means that there is resampling of the data, which introduces additional noise. This is where a lot of the noise in the data actually comes from, it seems, and less from the 8 bit conversion.
3. (?) It is implemented using a shader, and I'm not sure if the way the shader works introduces additional error, maybe due to resampling effects, or other graphics pipeline artefacts?

Point 1) could be solved by writing a custom export function (either in JS or server side, either is possible) - but this would take some time. I may have a look at this and see if a quick way is possible in browser (which would mean not having to pay for the expensive hosting/processing costs) however, writing a 16 bit PNG implementation for combining all this data in a fast and memory efficient way will be difficult whilst supporting all UE5 image sizes.

Point 2) could be fixed by only allowing zoom levels that display the tile data at native resolution. I'm not sure if this means just allowing integer zoom levels or not.

It would be a lot more convenient if one was able to just export from the heightmapper tool.

[drakgoku]

(and how would that work? Semitransparent landscape?)

The alpha channel in a heightmap is not essential, but it can be useful in some cases. A heightmap represents the height or elevation of a terrain in an image, and can be represented using only a grayscale information channel.

However, some terrain creation programs may use the alpha channel to store additional information, such as texture, moisture, vegetation, etc. In these cases, the alpha channel is important.

In short, the importance of the alpha channel in a heightmap depends on its intended use. If only elevation information is needed, a grayscale channel is sufficient. But if additional information is required, then the alpha channel can be useful.

Point 1) could be solved by writing a custom export function...

It looks good, you can do a pullrequest with that functionality and if the "tangrams" accepts it, that would be nice. Something like a button and it will run your PHP and download it.

[manticorp]

@drakgoku @llleeexxxoooppp

I have created a version of this that runs in-browser, you can find it here:

https://manticorp.github.io/unrealheightmap/

Let me know how you get on! Should be a lot easier than running it from the command line.

[llleeexxxoooppp]

@drakgoku @llleeexxxoooppp

I have created a version of this that runs in-browser, you can find it here:

https://manticorp.github.io/unrealheightmap/

Let me know how you get on! Should be a lot easier than running it from the command line.

I tested on Ubuntu 22.04 with Firefox 110.0 (64-bit) and it works time to time, i have two results: it works fine and even greatly more faster than previous console method, tiles are downloaded very fast and final image generated for several seconds, or something going wrong...

i've tried different resolutions and zoom options but this can also happen with a minimum resolution or a smaller zoom.
If it works it works great, but sometime I can't reach the server or something.

[manticorp]

@drakgoku @llleeexxxoooppp
I have created a version of this that runs in-browser, you can find it here:
https://manticorp.github.io/unrealheightmap/
Let me know how you get on! Should be a lot easier than running it from the command line.

I tested on Ubuntu 22.04 with Firefox 110.0 (64-bit) and it works time to time, i have two results: it works fine and even greatly more faster than previous console method, tiles are downloaded very fast and final image generated for several seconds, or something going wrong... i've tried different resolutions and zoom options but this can also happen with a minimum resolution or a smaller zoom. If it works it works great, but sometime I can't reach the server or something.

Aha - This should be fixed in the latest version. It is because the Longitude was overflowing (going below -180) - the Leaflet map doesn't clamp the value!